Bulk Export

Overview

interopiO supports version 1.0 of of the bulk export spec as described here .

 

Bulk Export Data Data Flow Diagram.drawio.png

 

In the bulk export process you can export a Group, which results in all the patients associated with the group and related resources, being exported.

Creating a Group resource:

A Group resource can be created by POSTing a Group resource json containing the required Patients. An example is shown below with the reference being Patient/ followed by the patient fhir id. 

{
"resourceType": "Group",
"id": "102",
"text": {
"status": "additional",
"div": "<div xmlns=\"http://www.w3.org/1999/xhtml\">\n <p>Selected Patients</p>\n <ul>\n <li>Patient Donald DUCK @ Acme Healthcare, Inc. MR = 654321</li>\n <li>Patient Donald D DUCK @ Acme Healthcare, Inc. MR = 123456</li>\n <li>Patient Simon Notsowell @ Acme Healthcare, Inc. MR = 123457, DECEASED</li>\n <li>Patient Sandy Notsowell @ Acme Healthcare, Inc. MR = 123458, DECEASED</li>\n </ul>\n </div>"
},
"type": "person",
"actual": true,
"member": [
{
"entity": {
"reference": "Patient/pat1"
}
},
{
"entity": {
"reference": "Patient/pat2"
}
},
{
"entity": {
"reference": "Patient/pat3"
}
},
{
"entity": {
"reference": "Patient/pat4"
}
}
]
}
 
 

Creating an App for getting access to the gateway:

The Group resource needs to be created in a CDR. You will need a CDR and a gateway that is attached to that CDR. You can use client credentials authentication to POST a group resource to the CDR gateway.

Create an app with security as shown below. You can specify any client secret that you wish.

Screenshot 2023-12-18 at 12.14.15 PM.png

Go to the app activations tab and create an activation on the CDR gateway with a public client.

Generating token to POST to the CDR gateway:

Append the client secret to the client id from the previous step, separated by a colon - an example is 941735e9-8e67-4378-9cc7-883f830bf300:secret. Base 64 encode this string using a website like https://www.base64encode.org/. You will get a string like OTQxNzM1ZTktOGU2Ny00Mzc4LTljYzctODgzZjgzMGJmMzAwOnNlY3JldA==.

You will need to use this base 64 encoded string to get an access token.

curl --location --request POST 'https://interopio.com/av2/oauth2/token?grant_type=client_credentials' --header 'Authorization: Basic OTQxNzM1ZTktOGU2Ny00Mzc4LTljYzctODgzZjgzMGJmMzAwOnNlY3JldA==' 

This will return an access token in the response.

{
"access_token": "eyJqa3UiOiJodHRwczovL2ludGVyb3Bpby1xYS5tY2tlc3Nvbi5jb20vYXYyL29hdXRoMi9qd2tzIiwia2lkIjoiMGZhZGY2ODAtODgwZi00OTk5LTk4OTgtNDdhMzUwM2ZiNWIwIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJ0ZXN0L29udGFkYS8qL2Q3MWExZjZlLTg4M2UtNDY4NS05Y2QzLTQ0ZjhjYmM0YTE4My9nMi1wYXRpZW50LWltcG9ydC9mYWxzZSIsImF1ZCI6ImQ3MWExZjZlLTg4M2UtNDY4NS05Y2QzLTQ0ZjhjYmM0YTE4MyIsIm5iZiI6MTcwMjkyNzY3NSwiaXNzIjoiaHR0cHM6Ly9pbnRlcm9waW8tcWEubWNrZXNzb24uY29tL2F2MiIsImV4cCI6MTcwMjkyOTQ3NSwiaWF0IjoxNzAyOTI3Njc1fQ.OOITBwiHzJSr5wuDLlqv2QS467cRAbdJ36gFohusRp25pjCOoefmrfkwNAUKPApUp7OCKrQq8PU99xa1b93qJgEUCebaOJ8jDWb9HCMlJKo2A1Th1MWOyHwOjloX6iG2uqQBEzpcmORKFvot37u9jdEgTF-lr-c7TjlvYIcQdc-UHoeW1VMly8OtkRVgsqwWFO2_RZHJEyZrwFOvdWz2ZYVoUieDE7rEbQ8Re8zfmLg6VmKCAtoIZFytUOVI4Tf0nOszynHdH5Xv47NceYMWSqyIRlL9AOoSbLyU59Gpalfg3Vf7boowOV1kgvh3a1GUhmZPFjcYHzRBDEaM_IvKkQ",
"token_type": "Bearer",
"expires_in": 1799
}

This token can be used as a bearer token to POST a resource to the CDR gateway.

Bulk Export Authorization:

Bulk Export requires SMART Backend Services Authorization. interopiO supports an out of band registration of the bulk client app with our authorization server. 

Registration of an app requires a client ID, a JWKS URL for the public key, scopes and the token signature signing algorithm (either RS384 ). After an app is registered, the app can make a call against the authorization server with a signed JWT and it gets a token which can then be used to kick off the bulk export process.

Bulk Export Process:

We have currently implemented bulk export for a Group resource that contains a list of Patients.

The export process is started by issuing an HTTP GET request for /Group/{Group Id}/$export against a gateway. An HTTP status of 202 Accepted is returned along with a Content-Location header that contains a polling location URL.

The bulk export process is started as a background process and begins with retrieving the Group resource that contains a list of Patients. Each Patient in the Group is retrieved along with its associated USCDI resources and saved into NDJSON files. Each resource type will be in a separate NDJSON file and the number of resources in a file is limited to 100. So there could be multiple files for a single resource type.

A polling location URL can be sent an HTTP GET request to query the status of the bulk export process. A 202 Accepted status will be returned while the bulk export is still running. The polling location URL will return an X-Progress header containing a count of the Patient resources being processed. When the export is finished, a 200 OK status will be returned along with an array containing the resource type and URL from which the exported file can be downloaded in NDJSON format.

Bulk Export API:

Pre-Register with interopiO Authorization server

This is an offline process where the bulk export client is pre-registered with the auth server. The public key for the client and scope (a string containing allowed operations) are given to the auth server as part of pre-registration and a client id is generated.  

Group Bulk Export Authorization Request

General Information

Method: HTTP POST

Endpoint: https://interopio.com/av2/oauth2/token

Parameters: these are to be sent as a string in the request body

client_assertion: a JWT signed using the client's private key and containing the following payload

{
"iss": "<client id assigned to bulk export client as part of pre-registration>",
"sub": "<client id assigned to bulk export client as part of pre-registration>",
"aud": "https://interopio.com/av2/oauth2/token",
"exp": <expiration time>,
"jti": "<jwt id as a random string>"
}

client_assertion_type: The following static value

urn:ietf:params:oauth:client-assertion-type:jwt-bearer

grant_type: The following static value

client_credentials

scope: The scope agreed with the auth server as part of pre-registration

Here is a sample parameter string with line breaks added for clarity

client_assertion=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzM4NCIsImtpZCI6ImI0MTUyOGI2ZjM3YTk1MDBlZGI4YTkwNWE1OTViZGQ3In0.eyJpc3MiOiI5NjQxN2M1ZS01NjVjLTQ2MGUtYmZiYy0yM2JhZTIwMDY0ODkiLCJzdWIiOiI5NjQxN2M1ZS01NjVjLTQ2MGUtYmZiYy0yM2JhZTIwMDY0ODkiLCJhdWQiOiJodHRwczovL2ludGVyb3Bpby1kZXYuY29tL2F2Mi9vYXV0aDIvdG9rZW4iLCJleHAiOjE2NjYxMDk4OTYsImp0aSI6IjM1NmMyZWI4NjEzMzI0NmZhZTc2NDViZmZjNTZjNGIwNGIwN2UyODZjZmRiMTdiM2Q4NzNkMmM3ZjU0ZDM0MDcifQ.fiZQl2RrLWzjH5rSFvDnAYSJX1dNFbBk4oGWSmvGU-eXSjnsVTw7J0f2MWMetZeuWRTsH0OHrRF99tzSehq3eH3Z7S3AHCLO4f9r6HDVrKelfpIpf9hbVYgEOhP77DiCWeikL_-UPB9q_6JVZK9_mHNfGbJXvqn-opgzOnDerLifHTc9Ef8-3tx_qaX4pfx7UGDH4zuKFWVm4KyzriNADlnQ8UG9TV8wmPyp153-FRX83_900ONIEwI4pl9mPnrt3GcIZejE88_cJ35jM-Ckczb_zSJkI2LpPqtpACbsuRksUF06QleL0cMBkPTFSg3LeEKzIxNDxWUBOeZxRVyByA
&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer
&grant_type=client_credentials
&scope=system%2F%2A.read

Supported Web Service Type: REST

Authorization: client_assertion parameter

Description

The bulk export client will call the token endpoint and obtain a JWT token that will be used in the kick-off process. 

Request

Request Headers

content-type: application/x-www-form-urlencoded

accept: application/json

Authorization: signed JWT sent as a parameter in HTTP POST request body

Response

Response Status

HTTP status code 200 OK if the request is valid

Response Body

A sample response is shown below

{

  "access_token": "<jwt access token>",
  "scope": "system/*.read",
  "token_type": "Bearer",
  "expires_in": 599
}

Group Bulk Export Kickoff

General Information

Method: HTTP GET

Endpoint: <FHIR Server URL>/Group/{GroupId}/$export?_since=yyyy-MM-dd&_type=<comma separated resource type list>

Supported Web Service Type: REST

Authorization: jwt (signed jwt from the authorization request)

Description

The bulk export kickoff requests starts the group bulk export process and returns a polling location URL that can be used for tracking the status of the bulk export process while it is running and for getting the URLs for the resources that can be downloaded, after the process is complete.

Request

Request Headers

Prefer: respond-async

Accept: application/fhir+json 

Authorization: Bearer JWT

Request Parameters

The request parameters are optional

Parameter Format Usage
_since

Observation resources will be included in the response if their state has changed after the supplied date

The date should be in the format yyyy, yyyy-MM or yyyy-MM-dd

examples are 2024, 2024-01, 2024-02-11

We currently support only the Observation resource for this parameter. In the examples given, only observations updated after 2024-12-31, 2024-01-31 and 2024-02-11 will be exported.
_type Comma separated list of resource types to be exported. For example Patient,AllergyIntolerance,DiagnostiocReport The Patient resource type needs to be included in the list

Response

Response Status

HTTP status code 202 Accepted if the request is accepted

HTTP status code 500 Internal Server Error if a bulk export other than a group export is specified

Response Headers

content-location: <FHIR Server URL>/$export-poll-status?_jobId=<jobId>

Response Body

A response body is returned if there is a kickoff error or when an unsupported bulk export is attempted to be kicked off. 

{
"timestamp": "2022-10-14T22:47:58.691+0000",
"status": 500,
"error": "Internal Server Error",
"message": "Only group export is currently supported using /Group/{GroupId}/$export",
"path": "/gateway/fhir/test/ontada/ikm21stcenturycure/$export"
}

Bulk Export Status

General Information

Method: HTTP GET

Endpoint: contained in the content-location header in the Bulk Export Kickoff response

Supported Web Service Type: REST

Authorization: jwt (signed jwt from the authorization request)

Description

The bulk export status endpoint can be called for checking the status of the bulk export process.

Request

Request Headers

Accept: application/json 

Authorization: Bearer JWT

Response

Response Status

HTTP status code 202 Accepted for a bulk export process that is in progress

HTTP status code 200 OK for a bulk export process that has completed

HTTP status code 404 Not Found if the _jobId parameter in the request is not found. This can happen if the content-location URL is old or if the request has already been cancelled

HTTP status code 500 Internal Server Error for a bulk export process that failed to process

Response Headers

The x-progress and retry-after headers are only returned in the case of a request that is still in progress. The former tells you how many patient ids have been included for processing and the latter tells you how many seconds to wait before sending another status request.

x-progress: <n> ids have started processing

retry-after: <n>

The content-type header is sent for a bulk export process that has finished.

content-type: application/json;charset=UTF-8 

Response Body

No response body is sent when the HTTP response status is 202. When the HTTP status is 200, the response body has the following format.

{
  "transactionTime": "2022-10-14 16:02:52:532 +0000",
  "request": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/Group/testgroup1014/$export",
  "requiresAccessToken": true,
  "output": [
    {
      "type": "Patient",
      "url": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/bulk/Group/c5082a22-0e03-4218-92c3-330bdd690e50"
    },
    {
      "type": "Condition",
      "url": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/bulk/Condition/da46d4f6-0e3c-46a7-8954-2dffea5b6155"
    },
    {
      "type": "DiagnosticReport",
      "url": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/bulk/DiagnosticReport/8e84f9de-0fa9-4bca-bec7-ae50d0d41175"
    },
    {
      "type": "Location",
      "url": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/bulk/Location/bfea2b97-0cad-49e2-9b83-529ea95a7d88"
    },
    {
      "type": "Observation",
      "url": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/bulk/Observation/92f6482c-52a3-40e2-bd45-f49d262a085f"
    },
    {
      "type": "Provenance",
      "url": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/bulk/Provenance/e96db5a5-a443-4712-bb14-7011f9e731fe"
    }
  ],
  "error": [
    {
      "type": "OperationOutcome",
      "url": "https://interopio-dev.com/gateway/fhir/test/ontada/ikm21stcenturycure/bulk/OperationOutcome/fa621fad-0126-4f78-93fc-3cf20fb11a66"
    }
  ]
}
Response body elements
Element Child Element Description
transactionTime   The time when the response was generated
request   The bulk export kickoff request
requiresAccessToken   If an access token is required to download an output or error file
output   An array of resource elements
  type an array element entry that tells you the type of the resource that can be downloaded
  url The resource url
error   An array of error elements
  type The type will always be OperationOutcome
  url The url for the error file

Bulk Export Cancel

General Information

Method: HTTP DELETE

Endpoint: contained in the content-location header in the Bulk Export Kickoff response

Supported Web Service Type: REST

Authorization: jwt (signed JWT from the authorization request)

Description

The bulk export process can be cancelled by sending an HTTP DELETE request to the polling URL. After a bulk export has been cancelled, a subsequent request to the polling URL will return a 404 Not Found HTTP status.

Request

Request Headers

accept: application/json

Authorization: Bearer JWT

Response

Response Status

URL status code 202 Accepted if the request is accepted

URL status code 404 Not Found if the _jobId parameter in the request is not found. This can happen if the content-location URL is old or if the request has already been cancelled

Response Headers

None

Response Body

No response body is sent.

Bulk Export File Download

General Information

Method: HTTP GET

Endpoint: a resource URL specified in the bulk export status response 

Supported Web Service Type: REST

Authorization: JWT if required

Description

A resource file can be downloaded using the resource URL. If the requiresAccessToken property described above is true, then a bearer JWT will be required.

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.