CanDIGv2 Architecture

Repo structure

Simplified repo structure with relevant files.

• Directoryetc/

  • Directoryenv/

    • example.env Copy and edit to configure variables
    • nightly_env.sh
  • Directorytests/ Integration and performance tests

    • test_integration.py Integration tests
  • Directoryvenv/ script, requirements for candig venv

    • activate.sh Activate candig conda env
    • requirements.txt Dependencies for the stack
• Directorylib/ submodule directories, docker compose, setup scripts and repos

  • …
• Makefile
• create_service_store.sh
• nightly_build.sh Script that can deploy stack nightly
• nightly_build_token.py Post nightly results to Slack
• post_build.sh Runs after to build to check running containers
• pre-build-check.sh Runs before build and checks dependencies
• settings.py Gets env variables from .env
• setup_hosts.sh Sets up hosts if LOCAL_IP_ADDR not set
• site_admin_token.py

List of Services and Components

The following table lists the individual repos for each service and helper library developed by the CanDIG team that contribute to the CanDIGv2 stack.

| Service/Component Name | Source | Description | |---------------------------|-----------------------------------------------------------------------|------------------------------| | authx | candigv2-authx | Library to facilitate interacting with AuthZ/AuthN services, Keycloak, Tyk, Opa, Vault & Access to minIO S3 objects | | CanDIG Data Portal | candig-data-portal | Front-end User interface for CanDIG Services | | CanDIGv2 Ingest Service | candigv2-ingest | Ingests clinical and genomic data into the CanDIG infrastructure. | | Clinical ETL Code | clinical_ETL_code | Code to convert spreadsheet format into the MoH data model in preparation for ingest into katsu | | Federation Service | federation-service | Distributes requests across each federated node of the distributed infrastructure | | HTSGet | htsget_app | Implementation of GA4GH htsget API which ingests and indexes VCF files and stores GA4GH DRS objects for retrieval | | Katsu | katsu | Manages the clinical metadata in a PostgreSQL database | | CanDIG OPA | candig-opa | Manages role-based access policies | | CanDIG Query service | candigv2-query | Manages front-end querying of services | | CanDIG logging | candigv2-logging | Logging package to unify logging across all services|

As well as in-house developed services, the CanDIG stack relies on external software which is configured to work within the stack, configurations are found in the /lib folder for each software, these include:

| Service/Component Name | Role | |-----------------------------------------|--------------------------------------| | Keycloak | Authentication management | | minio | Object storage for genomic files | | OPA | Manages role-based access policies | | Tyk | API management and redirection | | Vault | Secret and password management |

Adding a new service

New services can be added under lib directory. Please refer to the template for new services README for more details. diagram