CanDIGv2 Architecture

Repo structure

Simplified repo structure with relevant files.

• Directoryetc/

  • Directoryenv/

    • example.env Copy and edit to configure variables
    • nightly_env.sh
  • Directorytests/ Integration and performance tests

    • test_integration.py Integration tests
  • Directoryvenv/ script, requirements for candig venv

    • activate.sh Activate candig conda env
    • requirements.txt Dependencies for the stack
• Directorylib/ submodule directories, docker compose, setup scripts and repos

  • …
• Makefile
• create_service_store.sh
• nightly_build.sh Script that can deploy stack nightly
• nightly_build_token.py Post nightly results to Slack
• post_build.sh Runs after to build to check running containers
• pre-build-check.sh Runs before build and checks dependencies
• settings.py Gets env variables from .env
• setup_hosts.sh Sets up hosts if LOCAL_IP_ADDR not set
• site_admin_token.py

List of Services and Components

The following table lists the individual repos for each service and helper library developed by the CanDIG team that contribute to the CanDIGv2 stack.

Service/Component Name	Source	Description
authx	candigv2-authx	Library to facilitate interacting with AuthZ/AuthN services, Keycloak, Tyk, Opa, Vault & Access to minIO S3 objects
CanDIG Data Portal	candig-data-portal	Front-end User interface for CanDIG Services
CanDIGv2 Ingest Service	candigv2-ingest	Ingests clinical and genomic data into the CanDIG infrastructure.
Clinical ETL Code	clinical_ETL_code	Code to convert spreadsheet format into the MoH data model in preparation for ingest into katsu
Federation Service	federation-service	Distributes requests across each federated node of the distributed infrastructure
HTSGet	htsget_app	Implementation of GA4GH htsget API which ingests and indexes VCF files and stores GA4GH DRS objects for retrieval
Katsu	katsu	Manages the clinical metadata in a PostgreSQL database
CanDIG OPA	candig-opa	Manages role-based access policies
CanDIG Query service	candigv2-query	Manages front-end querying of services
CanDIG logging	candigv2-logging	Logging package to unify logging across all services

As well as in-house developed services, the CanDIG stack relies on external software which is configured to work within the stack, configurations are found in the /lib folder for each software, these include:

Service/Component Name	Role
Keycloak	Authentication management
minio	Object storage for genomic files
OPA	Manages role-based access policies
Tyk	API management and redirection
Vault	Secret and password management

Adding a new service

New services can be added under lib directory. Please refer to the template for new services README for more details. diagram