Skip to content

How to Run EHR Integration Pipelines with VPN Support

This guide provides instructions on how to run EHR integration pipelines using our Docker-based setup, with optional VPN support for accessing secure data sources.

Prerequisites

  • Docker installed on your system
  • Access to the ELT ingest pipelines repository
  • VPN configuration and authentication details (if required)

Setup

  1. Clone the ELT ingest pipelines repository:
git clone https://github.com/TriallyAI/elt-ingest-pipelines.git
cd elt-ingest-pipelines
  1. Build the VPN-enabled Docker image: 
    docker build -f Dockerfile.vpn -t elt-ingest-pipelines:vpn .

Running Pipelines

With VPN Support

To run a pipeline with VPN support, you need to provide the VPN configuration and authentication as base64-encoded environment variables.

  1. Encode your VPN configuration and authentication:
echo -n "your_vpn_config_content" | base64
echo -n "your_vpn_auth_content" | base64
  1. Run the pipeline with VPN support:
docker run -e VPN_CONFIG="base64_encoded_config" \
           -e VPN_AUTH="base64_encoded_auth" \
           elt-ingest-pipelines:vpn \
           python /src/pipelines/product/medstreaming_sql_pipeline.py \
           load-entire-database --tenant example --site example

Replace base64_encoded_config and base64_encoded_auth with your actual encoded VPN details.

Without VPN

To run a pipeline without VPN (for data sources that don't require it):

docker run elt-ingest-pipelines:vpn \
           python /src/pipelines/product/medstreaming_sql_pipeline.py \
           load-entire-database --tenant example --site example

Available Pipelines

Here are some of the available pipelines you can run:

  1. Load entire database:
python /src/pipelines/product/medstreaming_sql_pipeline.py load-entire-database --tenant TENANT --site SITE
  1. Load select tables: 
    python /src/pipelines/product/medstreaming_sql_pipeline.py load-select-tables --tenant TENANT --site SITE

Replace TENANT and SITE with appropriate values for your use case.

Troubleshooting

  • If you encounter VPN connection issues, ensure that your VPN configuration and authentication details are correct and properly encoded.
  • For pipeline-specific errors, refer to the error messages in the Docker logs and consult the pipeline documentation for more details.

Security Note

Always handle VPN configurations and authentication details securely. Avoid committing these details to version control or sharing them in plain text.

For more information on specific pipelines or advanced configuration options, please refer to the pipeline documentation or contact the data engineering team.