Skip to content

How to develop locally

Setup

  • Refer to the repository’s README of elt-orch-pipelines for essential setup instructions to configure your environment.
  • The project uses Docker to automatically manage dependencies within the container

Initialize the database

You need to run database migrations and create the first user account. To do this run:

docker compose up airflow-init

Running Airflow

To kick off all the Airflow services run:

docker compose up

Now the Airflow UI should be available at localhost:8080. The default account has the login airflow and the password airflow.

To check that all containers are running and healthy run docker ps

CONTAINER ID   IMAGE                                  COMMAND                  CREATED         STATUS                   PORTS                    NAMES
5743b625fff2   elt-orch-pipelines-airflow-webserver   "/usr/bin/dumb-init …"   4 minutes ago   Up 3 minutes (healthy)   0.0.0.0:8080->8080/tcp   elt-orch-pipelines-airflow-webserver-1
968806a9744c   elt-orch-pipelines-airflow-worker      "/usr/bin/dumb-init …"   4 minutes ago   Up 3 minutes (healthy)   8080/tcp                 elt-orch-pipelines-airflow-worker-1
0c56f9e9db9a   elt-orch-pipelines-airflow-scheduler   "/usr/bin/dumb-init …"   4 minutes ago   Up 3 minutes (healthy)   8080/tcp                 elt-orch-pipelines-airflow-scheduler-1
a286fca8d295   elt-orch-pipelines-airflow-triggerer   "/usr/bin/dumb-init …"   4 minutes ago   Up 3 minutes (healthy)   8080/tcp                 elt-orch-pipelines-airflow-triggerer-1
427ac097c06a   postgres:13                            "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes (healthy)   5432/tcp                 elt-orch-pipelines-postgres-1
fee79f734a5a   redis:latest                           "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes (healthy)   6379/tcp                 elt-orch-pipelines-redis-1

Cleaning up

The best way to recover from issues is clean up the docker environment and start from scratch.

To do this run this on the root directory:

docker compose down --volumes --remove-orphans