Transferring Data to the DRC

Introduction

This part of the process steps through the procedure for uploading EHR data files to the Data and Research Center (DRC) Google Cloud Storage Buckets. The process applies to all program partner organizations that are responsible for sharing participant data (e.g., EHR) with the DRC. It details the required documentation and parameters needed as well as the testing required to upload EHR data to the DRC Google Cloud Storage Buckets.

• You must have a valid PMI-Ops account and provisioned access to your Google Storage Cloud bucket before following the instructions below. To request your access, please fill out the DRC User Access Request form.

Instruction on accessing Google Storage Cloud bucket

• Log in to your PMI-Ops account in your browser

• Open URL https://console.cloud.google.com/storage/browser/aouXXX 

  • aouXXX is your bucket name

• After you open your bucket, you can view/upload/delete folders in your bucket

• If you encounter permission issue, please contact DRC Support ( [e-mail] ) for help

Assumptions

• Site EHR data is in OMOP format.

• EHR files are to be uploaded as separate files - one file per table. The file name will be the name of the table and the files will adhere to the Comma-Separated Values (CSV) file format.

• For each submission, all eligible participants data for all tables will be submitted.

Step 1: Log out of any currently active Google IDs. Log into MyAccount (https://myaccount.google.com/) with the pmi-ops.org credentials provided by the DRC Support Team.

Step 2: Data Stewards will receive an email from the DRC Support Team containing a direct link to the assigned Bucket. Open a web browser and navigate to the Google Cloud Storage Bucket via the direct link shared with you by the DRC Support Team.

NOTE: If you receive an error message that your account is inactive, please email [e-mail] to re-activate your account.

Step 3: Ensure your submission files have all been placed in an appropriately named folder in preparation for submission. All files must be submitted in a folder adhering to following naming convention : [YYYY-MM-DD-v#]. Below are examples of how the convention is to be used. Data files for each upload cycle must be contained within unique folder delineated by the version number at the end of the file name. Loose files uploaded to the bucket root will not be accepted.

For each files, use lowercase and underscore for space in naming your files. For example, the procedure occurrence table will be named procedure_occurrence.csv.

Standard Naming Convention for Folder

• • • Example Day 1 Folder Names

      • 2018-04-13-v1

      • 2018-04-13-v2

    • Example Day 2 Folder Names

      • 2018-04-14-v1

      • 2018-04-14-v2

Step 4: Uploading with gsutil

• Open Google Cloud SDK shell.

• Authenticate: Enter gcloud auth login

  • This opens a browser window to authenticate to Google Cloud Platform. Login using the assigned pmi-ops account (e.g. [e-mail])

• Set project: gcloud config set project all-of-us-staging (for example)

• Upload files: gsutil -m cp -r [YYYY-MM-DD-V#] gs://[hpo-site-bucket]

  • Example: gsutil -m cp -r 2018-04-13-v2 gs://hpo-site-bucket

• Upload

NOTE: for instructions on installing gsutil click here.

The following is an alternative Upload Procedure to using gsutil

• In a desktop environment, navigate to the data set. Drag and drop the data set files from the local folder into the browser window where the Google Cloud Storage Browser is open to the folder created in Step 3 under Procedures.

Step 5: Within 3 hours of uploading data to a folder within the Bucket, additional files and folders will appear in the processed folder. This content is there to provide the data steward with useful information about the outcome of the upload and the some basic descriptive feedback about the contents of their payload/data files. You will also receive an email in your inbox with the results.html file attached.

• Results.html

  • This file contains a list of all files expected to be submitted to the DRC

  • For each file there are 3 states which indicate

    • Was the expected file found?

    • Was the expected file parsed?

    • Was the expected file loaded?

  • Warnings/errors

  • In the results.html, this will contain a list of file names that were submitted and the potential problems that were encountered while being processed. This document provides suggestions on how to interpret some of the errors.

  • Heel errors

  • If there are any Achilles heel errors, it will be shown in a table listing each errors

• Curation report folder

  • This folder contains reports of the submission within a single web page. To download and view the reports follow these instructions.

Issue Reporting

Report any issues with bucket access or PMI-Ops accounts to [e-mail]. A member of the DRC team will contact the Data Steward to assist with resolution of the issue. For other questions, please email EHR Ops team at [e-mail].