Adding new superset instances

All the installation documentation for superset can be found from: https://superset.incubator.apache.org/ https://superset.incubator.apache.org/installation.html

Superset dockers are located in attracs.backend.1. This server can be accessed with ssh henki@attracs.backend.1 .

GCP project

Use terraform to create project and 2 service accounts

create airbyte and superset service accounts.

Database

MySQL

Create MySQL Cloud SQL database in GCP. This can be found in NO ORG ➝ Attracs Services project.

Create database:

Database Name: superset_customerName
Character set: utf8
collation: utf8_general_ci

Add user account:

Select -> Built-in authentication
Username: superset_customerName
Password: generate 40 char pass
host name: Allow any host (%)

Superset Docker container

Define port in https://docs.google.com/spreadsheets/d/10trjuvrp_N8-OrqwsHchAysTIcrJRNqZ/edit#gid=878011441

Next deployment for superset, Not in use Check the new way of deploying managing superset.

Making new docker can be done from attracs-linux-playbooks.

Add new requirements.txt for new customer docker/superset/dockerfiles in attracs-linux-playbooks.

Add new dockerfile for new customer docker/superset/dockerfiles in attracs-linux-playbooks.

Add new custom configs for new customer docker/superset/config in attracs-linux-playbooks. These need to be encrypted. There is 2 custom config files named:

custom_sso_security_manager_customername.py Set this line in:

class CustomSsoSecurityManager(SupersetSecurityManager):
roles = me['https://superset-customerName.attracs.com/roles']

superset_config-customername.py These need to be set

SUPERSET_WEBSERVER_PORT = PORT_NUMBER
SECRET_KEY = 'GENERATE YOUR OWN VERY SECRET KEY'
SQLALCHEMY_DATABASE_URI = 'mysql://superset_customerName:PASSWORD@35.228.155.85:3306/superset_customerName'
'client_id': 'Client_id_from_auth0_app'
'client_secret': 'Client_secret_from_auth0_app'

Adding new docker container you will need to add few things to docker-superset-build-and-run.yml .

need to add new vars for customer:

  vars:
    - deploy: ""
    - ansible_python_interpreter: "/usr/bin/python3"
    - superset_config_path: /etc/superset-docker{{deploy}}
    - superset_port: 4064
    - superset_version: "1.5.0"
    - superset_version_nightly: "1.5.0"
    - supserset_version_customer1: "1.5.0"
    - supserset_version_customer2: "1.5.0"
    - supserset_version_customer3: "1.5.0"
    - supserset_version_customer4: "1.5.0"
    - supserset_version_customer5: "1.5.0"
    - supserset_version_customer5: "1.5.0"
    - superset_translation_path: /etc/superset-translations

Add new port definition:

  - name: define port for nightly
    set_fact:
      superset_port: 5064
    when: deploy == "-nightly"

  - name: define port for CustomerName
    set_fact:
      superset_port: PORT_NUMBER
    when: deploy == "-customerName"

Add new define version:

  - name: define version for nightly
    set_fact:
      superset_version = superset_version_nightly
    when: deploy == "-nightly"

  - name: define version for CustomerName
    set_fact:
      superset_version = superset_version_CustomerName
    when: deploy == "-CustomerName"

When running docker-superset-build-and-run.yml you need to define on the vars deploy: "" section what customer docker you want to deploy.

After running build and run script run:

ansible-playbook docker/docker-superset-db-upgrade.yml --vault-password-file=.vault_password_file

This script runs db upgrade and init for superset.

Auth0 configs

First add new auth0 application from the applications ➝ applications

Name for the application: Customername Superset Select regular web application

In the new application configs go to settings ➝ application URIs

Allowed Callback URLs: https://superset-customername.attracs.com/oauth-authorized/auth0

Allowed Web Origins: https://superset-customername.attracs.com

Remember to save changes at the end.

Create new Role

Name: supersetCustomerName_Admin Description: CustomerName superset admins

Add users to new role. Most of the time you need to add yourself and Hannu Rauhala.

Next add new rules set

Go to Auth Pipeline ➝ Rules

Create new rule: Name: Add supersetCustomerName roles Change Client_ID_auth0 ➝ to your auth0 application client ID if (role.startsWith("supersetCustomerName_")) { let supersetRole = role.substr("supersetCustomerName_".length);

function (user, context, callback) {
  const namespace = 'https://superset-CustomerName.attracs.com';
  const assignedRoles = (context.authorization || {}).roles || [];
  if (context.clientID === 'Client_ID_auth0') {
    let supersetRoles = [];
    assignedRoles.forEach(role => {
      if (role.startsWith("supersetCustomerName_")) {
        let supersetRole = role.substr("supersetCustomerName_".length);
        if (supersetRole.length > 0)
          supersetRoles.push(supersetRole);
      }
    });

    if (supersetRoles.length > 0) {
      let idTokenClaims = context.idToken || {};
      let accessTokenClaims = context.accessToken || {};

      idTokenClaims[`${namespace}/roles`] = supersetRoles;
      accessTokenClaims[`${namespace}/roles`] = supersetRoles;

      context.idToken = idTokenClaims;
      context.accessToken = accessTokenClaims;
    }
    }
  callback(null, user, context);
}

DNS

Add new superset instance to dnsmade eazy with:

./gcp/scripts/dns/create_cname_record.sh superset-customerName frontend

Add new frontend configuration to frontend/frontend-data.yml

  - {
      subdomain: superset-CustomerName,
      backends: [
        'fi.1 {{ attracs_backend_1 }}:PORT_NUMBER'
      ],
      x_forwarded_proto_header: true,
      healthcheck: /health
    }    

after adding new one run: ./scripts/frontends-update-configs.sh