Search

Import File Via AWS S3 or SFTP

Rebekah Garner
Rebekah Garner
  • Updated

Claravine's platform, The Data Standards Cloud, can be connected to an AWS S3 Bucket or SFTP Folder and any new files dropped there will be automatically imported as a submission on that template. If the file follows the correct governance applied per the template, the platform will import the file and submit it in viewable batches of 10,000 rows, assign Pattern key, validate field input values, generate and validate the URL, and disperse the data to all output configuration destinations.

To import a file via AWS S3 or SFTP:

  1. An Admin or Manager with Account permissions must connect The Data Standards Cloud to the designated AWS S3 Bucket or SFTP Folder. Read more about connecting an Account.

  2. An Admin or Manager with Template permissions must add the Inbound Integration for AWS S3 or SFTP on the Template. Read more about adding AWS S3 Inbound Integration or SFTP Inbound Integration on Templates.

  3. It is recommended to use the provided Template File to ensure your submission file contains the correct headers.  On the Datasets Page, find the Template and click the Import File  action icon.


  4. Click Download Template File in the Import CSV File window.
    mceclip0.png

  5. Open the CSV Template File from your computer’s Downloads folder. Enter or paste the information in the Template File, up to 50,000 rows of data.


    Field governance set on the template needs to be met in the uploaded Template File. For example, value must be in a pick list, no spaces allowed, character limits, etc. If you attempt to upload a file with values that do not match the template governance, your submission will be paused and you will be instructed to correct the errors.

    Do not delete any column fields from the Template File. If an input value is optional, leave the field blank.

    mceclip0.png Claravine's platform is case sensitive, if a value is in a pick list, it must match the case exactly.

  6. Save the file:
      1. The name of the file will become the Submission Name.
      2. Your Administrator may require a specific File Name Prefix if using AWS S3.
      3. Save As CSV UTF-8 file type.

  1. In AWS S3 or SFTP: Drag the saved Template File or click Upload and browse to find the saved file.

  2. As soon as the file is dropped into the folder, the platform begins importing, processing, and validating the data in the Template File you loaded in two steps: 
      1. Field validation to ensure the imported data follows the requirements and governance designated on the template.
      2. Link validation.

  1. Log in to Claravine's platform to see the status in the Submission menu. If you do not see your file processing, there was an error in your data. Please review and correct the data and upload it to AWS S3 or SFTP again.

Files are ingested in batches of 10,000 rows. The Submission Name indicates the batch of 10,000 {Saved File Name}{Batch}


Claravine's platform shows the Processing Submission progress bar to indicate the state of your imported file. Your data will automatically be available as soon as the processing has finished.

There are processing five steps:

  1. Created - the Import File process is initiated by clicking Upload
  2. Importing - the Template File is being imported
  3. Processing - checking file formatting, field columns match the template
  4. Validating Rules - validating governance and restrictions of data
  5. Exporting - sending data downstream, emailing confirmation, making the data viewable



If the Import is Successful

If the import is successful and there are no errors found in your file, the status of the submission will show a green checkmark and Claravine will send an email to the submitter notifying them of their “Successful Submission”.


If the Import Fails Due to Errors Found in Your File

If the import fails due to errors found in your file, you will not see a warning status on the Submissions list, you will not see your submission at all.

  1. Go back to your template file and make the corrections needed. The data in the file follows the exact same governance and restrictions as the template in the UI.

  2. Confirm you are using the most up to date downloaded template file with the correct headers.

  3. Confirm you have saved your file as CSV UTF-8.

  4. Repeat steps 7 and 8 again and confirm receipt and processing in Claravine's platform.

Related Resources

Comments

0 comments

Article is closed for comments.