Update Production CanDIG

As new stable versions of the CanDIGv2 stack are released, it is recommended that production deployments be updated as soon as possible as deployments at different versions are unlikely to be able to be federated.

The process for updating depends on the changes contained in a particular release. The release notes for a particular release should indicate whether an existing deployment needs a full reinstall or can be updated without deleting any existing data. The steps in the process should follow the following general steps:

Step-by-step guide to updating CanDIG

Backup your existing deployment using the steps on the page: Backing up and restoring CanDIG data. This should be done regardless of wheteher you think the deployment will destroy any data, just in case anything goes wrong. Things you will want to back up are:
- ingested genomic and clinical data - Guide here: Back up postgres databases
- log files - Guide here: backup logfiles
- user authorizations - Guide here: backup secrets and authorizations
- federation details (if your node is federated with other nodes) - see expandable section below:
  How to save currently stored federation information
  1. Get a site admin token and in a terminal session, store in a variable called TOKEN, store your candig domain in a variable called CANDIG_URL
  2. Call the GET method to list all currently federated servers:
    Terminal window
    curl -X 'GET' \ $CANDIG_URL'/federation/v1/servers' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer '$TOKEN
  3. Save the output to a text file so you have the information available after you have upgraded your deployment
  If a node you are federated with has performed a ‘destructive rebuild’ which reset their keycloak instance, you may need to ask them for a new bearer token.
- git repos diffs - see expandable section below:
  Example bash script of how to save git diffs of all CanDIGv2 repos
  #!/bin/bash current_date=$(date +%F) # Set the output file path in the tmp/logs directory output_file="tmp/logs/diff-$current_date.diff" # Create the tmp/logs directory if it doesn't exist mkdir -p tmp/logs # Clear the output file if it exists > $output_file # Get the diff of the main repo and append to the file echo "Diff for the main repo:" >> $output_file git diff --no-ext-diff --submodule=diff >> $output_file echo "---------------------------------------------------" >> $output_file # Loop through each subrepo in the 'lib' directory for dir in lib/*/; do if [ -d "$dir/.git" ]; then # Enter the subrepo directory and get the diff echo "Diff for subrepo $dir:" >> $output_file (cd "$dir" && git diff) >> $output_file echo "---------------------------------------------------" >> $output_file fi done echo "Git diff for main repo and subrepos has been saved to $output_file."
Pull the latest stable release from the CanDIGv2 repo and update submodules
Terminal window
```
cd CanDIGv2
git checkout stable
git pull
git submodule update --init --recursive
```
Ensure the candig conda environment is activated and update to the latest requirements
Terminal window
```
conda activate candig
pip install -r etc/venv/requirements.txt
```
Backup your current .env and modify the latest example.env
Terminal window
```
cp .env .env-bak
cp etc/env/example-production.env .env
```
Use the backed up copy to enter any deployment specific information

You can use a file comparison tool or use the bash diff command to see any changes between the two files
Apply any other deployment specific changes such as those found when performing a git diff in step 1
If the release allows for a non-destructive rebuild, use the rebuild-keep-data target to clean and recompose services without destroying any data. It is still recommended you back up all data before proceeding with this method.
Terminal window
```
make rebuild-keep-data
```
If you have removed the default site admin, you will need to enter your admin credentials when prompted by the build process. Be sure to pay attention in the terminal, the build will be stuck until you enter the credentials.

This target will clean, build and compose services but will not delete any ingested data, user authorization information or logging information.

Proceed to step 9.
If the release needs a full rebuild, all services will need to be cleaned, rebuilt and recomposed and any existing data reingested or restored from backups. This is done by running the following commands

This destroys all data in the system so it is important to ensure there is a backup of all existing data before proceeding with this method.
Terminal window
```
make clean-all
make build-all
```
If you performed a destructive rebuild, it will be necessary to reingest and restore backed up data following the guide. This includes any clinical and genomic data as well as information about user authorizations. Federating to other nodes will also need to be reinitiated with all other nodes in the network. Federating with nodes at a different stable version may not be possible.

Depending on the changes that were made, it may not be possible to directly restore from backup, e.g. if the underlying structure of the postgres databases has changed, the data will need to be reingested through ingest, rather than restored directly to the database.
Run integration tests to ensure all the services are running as expected

If you have removed the default site admin, integration tests will not work see expandable section for instructions
Running integration tests when default site admin is removed
If you revoked site admin rights from site_admin@test.ca you can temporarily reinstate rights to run the tests and then revoke them again
1. To reinstate site_admin@test.ca as a site admin user, from the CanDIGv2 repo in a terminal
First get a token:
Terminal window
```
source env.sh
```
Terminal window
```
CURL_OUTPUT=$(curl -s --request POST \
--url $KEYCLOAK_PUBLIC_URL'/auth/realms/candig/protocol/openid-connect/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data grant_type=password \
--data client_id=$CANDIG_CLIENT_ID \
--data client_secret=$CANDIG_CLIENT_SECRET \
--data username=<YOUR EMAIL ADDRESS> \
--data password=<YOUR PASSWORD> \
--data scope=openid)
```
Terminal window
```
export TOKEN=$(echo $CURL_OUTPUT | grep -Eo 'access_token":"[a-zA-Z0-9._\-]+' | cut -d '"' -f3)
```
1. Add the site_admin@test.ca back as a site admin:
Terminal window
```
curl -X POST $CANDIG_URL'/ingest/site-role/admin/user_id/site_admin@test.ca' -H 'Authorization: Bearer '$TOKEN
```
1. Uncomment the default site admin lines in the env.sh and .env file
If the default site admin password was deleted from env.sh, you can copy it from tmp/keycloak/test-site-admin-password
1. Run integration tests
Terminal window
```
make test-integration
```
1. Revoke site admin rights from the default site admin
Terminal window
```
curl -X DELETE $CANDIG_URL'/ingest/site-role/admin/user_id/site_admin@test.ca' -H 'Authorization: Bearer '$TOKEN
```
Terminal window
```
make test-integration
```
If tests fail, check out the troubleshooting section or reach out for help.

Reinstate federation with other nodes

Use the information in the file you created in step 1 to add all federated servers back to your deployment’s federated network. You will need to contact the site admin at each node for a token to re-establish the connection. Your calls to add each server will be something along the lines of the code block below, or see the federation README

curl -X 'POST' $CANDIG_URL'/federation/v1/servers' \
 -H 'Content-Type: application/json' \
 -H 'Authorization: Bearer '$YOUR_TOKEN \  # the token of the site admin at the site trying to add the server
 -d $'{ \
  "server": { \
    "id": "NODE1", \
    "url": "http://www.candig-node1.ca", \
    "location": { \
      "name": "NODE1", \
      "province": "ON", \
      "province-code": "ca-on" \
    } \
  }, \
  "authentication": { \
    "issuer": "https://www.candig-node1.ca/auth/realms/candig", \
    "token": $NODE1_TOKEN \  # any user token from the site you want to add, can be expired
  } \
}'