Skip to content

SecDevOps Documentation

Failed Update

Initializing search

SecDevOps Documentation

Welcome
Hetzner Meeting 25.6.2026 — CEO notes
Open file - Security Warning
City
City
DevOps
DevOps
- AholaDigitalplaybook quartz
- GCP Cloud Source Repositories
- Systems and Services template
- CICD
  CICD
  - CI/CD: Github and GCP Artifact Registry
- Docker
  Docker
  - Docker: how to save docker images
- Event, Product & Service notes
  Event, Product & Service notes
  - Google SRE Masterclass 2024
- Github
  Github
  - Adding external users to Github
- Incidents
  Incidents
  - 25.10.2023 - TMS Stable machine type update
  - Auth0 SendGrid SMTP authentication failure
    
    Auth0 SendGrid SMTP authentication failure
    
    Auth0 cannot send email through SendGrid — 451 Authentication failed: Could not authenticate
  - Linux VM failed update
    
    Linux VM failed update
    
    Incident Summary: GCP Patch Job Failure on TMS production server
    
    Incident Summary: GCP Patch Job Failure on Production Server
    
    Post update checks
  - Wazuh false positive
    
    Wazuh false positive
    
    Issue
    
    Runbook: Wazuh Rule 510 "Trojaned version of file detected" — Coreutils False Positives
- Internal Apps
  Internal Apps
  - Cargo Safety
  - HAProxy
  - Memcached
  - Databases
    
    Databases
    
    Connecting to a database
    
    Postgres
    
    Postgres
    
    Postgres: Installing a new extension
    
    Postgres PG_STAT_STATEMENTS
    
    Postgres: fix incorrect view table ownership
    
    Postgres: grant yourself superuser privileges
    
    Postgres Manipulate JSON rows
    
    Postgres Manipulate JSON rows
    
    Postgres: Query to remove a single JSON key from an array object
    
    Postgres: SELECT (list) a JSON-type row's keys separately
  - Docusaurus
    
    Docusaurus
    
    Docusaurus build crash
  - Manage
    
    Manage
    
    Manage: Troubleshooting users login issues
    
    Manage: re-activating a deleted external user
    
    Manage Creating a new organization
    
    Manage Creating a new organization
    
    Manage: Stable release
    
    Manage: TMS city configuration
  - Managevisor
    
    Managevisor
    
    Managevisor: Application configs
    
    Managevisor: Creating or updating managevisor information (servers)
    
    Managevisor: Deploying mobile applications
    
    Managevisor: Giving access to a new user
    
    Managevisor: Managevisor restart
    
    Managevisor: New app documentation (needs review)
    
    Managevisor: Starting and stopping services
    
    Managevisor New applications Revised documentation
    
    Managevisor New applications Revised documentation
    
    Managevisor: New applications: Legacy documentation
  - Superset
    
    Superset
    
    Adding new superset instances
    
    Adding superset users
    
    Superset installation
    
    Superset upgrade
    
    Updated superset (new deployment)
    
    Updated superset (new deployment)
    
    1. Ansible
    
    Steps to setup
  - Supervisor
    
    Supervisor
    
    Supervisor: Deploying applications
    
    Supervisor: OOM
    
    Supervisor: Organization-specific application configuration
    
    Supervisor: Updating Managevisor services
    
    Supervisor: Updating an application on host
    
    Supervisor: Updating list of applications on a host
    
    Supervisor errors
  - Valhalla & Vroom
    
    Valhalla & Vroom
    
    Known issues
- Issues & Problems
  Issues & Problems
- Operating Procedures
  Operating Procedures
- Third Party Apps
  Third Party Apps
  - Alteryx
  - DNS Made Easy
  - HERE maps
  - Slack create incoming webhook
  - Slack troubleshooting
  - Airbyte
    
    Airbyte
    
    Airbyte: Creating new table syncs
    
    Airbyte: modify last sync start time
    
    How to add new tables to an existing Airbyte sync
    
    How to make Airbyte connections
    
    How to re-sync fresh Airbyte data in case of e.g. corrupted or duplicate data
    
    Install update investigate
    
    Install update investigate
    
    Clean install
    
    airbyte update steps
    
    Clean Install AirByte 64.1 helm
    
    Clean Install AirByte 64.1 helm
    
    configurations
    
    Setup with ABCLT Currently in use
    
    Setup with ABCLT Currently in use
    
    plan for testing
  - Google workspace
    
    Google workspace
    
    Gmail
    
    Gmail
    
    Company group incoming restriction gmail
    
    DKIM setup for gmail
    
    DMARC setup for gmail
    
    Gmail footer append
    
    SPF setup for gmail
  - Syncthing
    
    Syncthing
    
    Syncthing: Configuring a new remote device for sharing and syncing
    
    Syncthing: Installation
  - e Mail
    
    e Mail
    
    Google: How to create a shared inbox
- Tools on Local
  Tools on Local
External Development
External Development
- Palbassador
- Infra
  Infra
- Tailorframe
  Tailorframe
  - Database sync to bigquery
- Moreutils
  Moreutils
  - moreutils — Research & Recommendation
IT
IT
- Office network (aholadigital wifi)
- License reporting
  License reporting
  - Arrow ArrowSphere
  - Crayon CloudIQ
- Servicedesk
  Servicedesk
Infra
Infra
- Cloud
  Cloud
  - Azure
    
    Azure
    
    Create Azure account
  - GCP
    
    GCP
    
    Cloud Armor
    
    Cloud Spanner
    
    Big Query
    
    Big Query
    
    Big Query
    
    GCP BQ: How to give view permissions to datasets
    
    Cloud SQL
    
    Cloud SQL
    
    GCP CloudSQL Automated Database Backups
    
    GCP CloudSQL: Changing the internal network of a database instance
    
    GCP CloudSQL Database Recovery
    
    GCP CloudSQL: Enterprise vs Enteprise Plus
    
    GCP CloudSQL: Exporting a database from CloudSQL
    
    GCP CloudSQL Manual Database backups
    
    GCP CloudSQL Read-replica instance
    
    GCP CloudSQL database connection
    
    GCP CloudSQL instance
    
    GCP CloudSQL Query Insights
    
    GCP CloudSQL Query Insights
    
    Cloud SQL Query Insights permissions
    
    GCP CloudSQL Active Query Analysis
    
    Confidential Compute
    
    Confidential Compute
    
    Confidential Computing VM
    
    GCP CC Integrity Monitoring failure modes
    
    Load Balancer
    
    Load Balancer
    
    GCP Load Balancer: HTTP-to-HTTPS redirect
    
    GCP Load Balancer: Migrating from classic to modern LB
    
    GCP Load Balancer Identity Aware Proxy (IAP)
    
    GCP Load Balancer Identity Aware Proxy (IAP)
    
    GCP LB IAP: Troubleshooting
    
    Ops Agent
    
    Ops Agent
    
    GCP Ops Agent Logging
    
    GCP Ops Agent: Troubleshooting
    
    GCP: Updating the agent
    
    PubSub
    
    PubSub
    
    Message Brokers (pub/sub)
    
    Virtual Machines
    
    Virtual Machines
    
    GCP Managed Instance Groups
    
    GCP: Reverting/rollback an instance to a snapshot
  - Hetzner
    
    Hetzner
    
    Hetzner Could Servers
    
    Hetzner configs
    
    Road
    
    Untitled Page
    
    City
    
    City
    
    Cost comparison between gcp and hetzner configuration
    
    Disk space clean up
    
    Disk space clean up
    
    hetzner-backend1 free disk clear
    
    hetzner-backend2 free disk space
  - Management
    
    Management
    
    GCP AO Plan
    
    GCP: Connecting to instances
    
    GCP projects
    
    IaC
    
    IaC
    
    Ansible
    
    AWX (Ansible)
    
    AWX (Ansible)
    
    AWX (Ansible): 4. AWX Configurations basics
    
    minicube - awx
    
    Archive old instruction for backup restore on new server and upgrading
    
    Archive old instruction for backup restore on new server and upgrading
    
    AWX (Ansible): AWX Install
    
    AWX (Ansible): AWX how to migrate from on-permise to hetzner
    
    AWX: AWX Screen shots before update
    
    AWX restart
    
    AWX (Ansible) 2. AWX Update steps
    
    AWX (Ansible) 2. AWX Update steps
    
    AWX (Ansible): 2. AWX Update steps: preparation
    
    AWX (Ansible) 3. AWX update steps after 17.0.0 minicube
    
    AWX (Ansible) 3. AWX update steps after 17.0.0 minicube
    
    AWX minicube - recreate
- Monitoring & Logs
  Monitoring & Logs
- On Prem
  On Prem
  - Hetzner on-prem
  - adi-devops-ci-instance-stable-1
  - devops.4
  - tms-backend-1b-nightly
  - Core servers
    
    Core servers
    
    Core servers: creating a new core server
  - Proxmox (A.I. on prem server)
    
    Proxmox (A.I. on prem server)
    
    Deploy Microsoft Defender Endpoint on Linux
    
    Install Tailscale on LXC
    
    Install Tailscale on workstation
    
    Setting VM/LXC automatic backups
  - Devops openvpn & devops ca machine details
    
    Devops openvpn & devops ca machine details
    
    Devops OpenVPN Installation process
  - Pg databases
    
    Pg databases
    
    pg-databases-stable
Integrations
Integrations
- Datastream -> Bigquery -> Power Bi
- AD Sympa
  AD Sympa
- Biztalk
  Biztalk
- Camel
  Camel
  - Adi-Camel
  - EDI Helper
  - Edi-helper nightly and edi-helper-grpc
  - SFTP service docker
  - adi-camel configs
  - Investigations
    
    Investigations
    
    To do 1v:
    
    To do final 2v:
    
    click up logs what I find about camel.
    
    stuff
- ClickUp
  ClickUp
- LogicApps
  LogicApps
- N8N
  N8N
  - N8N automated DevOps documentation backup
Road
Road
- Database Creation, Ownership, and Permissions Automation Guide
- Database migration
- AO
  AO
  - AO MSSQL Database restore (Azure)
  - AO Rclone
  - AO monitoring
  - AO online-api.attracs.com
  - AVD Login Instructions to Attracs Online
  - AO External
    
    AO External
    
    AO Patch
    
    AO update
  - AO GCP
    
    AO GCP
    
    Elisa Terraform onboarding
    
    AO GCP Environment
    
    AO GCP Environment
    
    AO Access
    
    AO Architect
    
    AO DB Validator
    
    AO Introduction
    
    AO Monitoring & Alerting
    
    Building AO
    
    Create AO RestServer and Load Balancer
    
    Create new host server
    
    Environment permissions
    
    Evolution script
    
    Update EdgeWebview2
    
    Windows license activation problem
    
    AO Clients
    
    AO Clients
    
    Ahola & Special
    
    Apps what client uses
    
    Externals
    
    AO File and Database backup
    
    AO File and Database backup
    
    DB Backup
    
    GCP AO File backups
    
    AO GCP Infrastructure
    
    AO GCP Infrastructure
    
    AO Cloud Armor
    
    AO Firewall
    
    AO Integrations
    
    AO Load Balancers
    
    AO Port Definitions
    
    AO Servers
    
    AO Update management
    
    Cloud SQL
    
    Cloud SQL
    
    AO Database datamasking
    
    AO Database optimization
    
    AO SQL Instances and Databases
    
    Cloud SQL Automated Database Backups
    
    Cloud SQL Manual Database backups
    
    SQL Database Recovery
    
    Policies
    
    Policies
    
    2-way trust
    
    GCP AD - Password policy
    
    RDS policies
    
    AO GCP Release Deployments
    
    AO GCP Release Deployments
    
    AO GCP Release - Deployments
    
    Release task template
    
    AO Logging, Monitoring, Alerting
    
    AO Logging, Monitoring, Alerting
    
    AO Logging
    
    AO-logs
    
    Resolving alerts
    
    ToDo List of AO monitoring and alerting
    
    Windows logs with OPS-Agent
    
    AO Patch
    
    AO Patch
    
    Automated build and patch (Ahola TIP)
    
    AO Problems
    
    AO Problems
    
    Client Problems with MAPS
    
    How to end user session and process when patching client
    
    Open .xml files in IE with client
    
    Reset AO user RDP-Profile
    
    Windows Protected your PC
    
    AO SMB Volume
    
    AO SMB Volume
    
    How to create and connect NetApp-Volumes
    
    AO VPN tunnel
    
    AO VPN tunnel
    
    GCP - Azure tunnel
    
    Attracs Online
    
    Attracs Online
    
    Datapump
    
    Datapump
    
    Licensing Server
    
    List of Datapumps
    
    Run Datapump
    
    User Profile Disk
    
    RDS Collections
    
    RDS Collections
    
    Can't publish RemoteApp
    
    Software
    
    Software
    
    AO Servers
    
    Client
    
    Distance Calculator
    
    Forklift
    
    Service ID & OSS
    
    Biztalk
    
    Biztalk
    
    Biztalk -> Camel SFTP
    
    Biztalk Integration Customs
    
    Biztalk Integrations Firewall & Cloud Armor configurations (Whitelist)
    
    Extend windows local disk
    
    Broker and GW's
    
    Broker and GW's
    
    RDP - Gateway connection handling
    
    PTVxServer
    
    PTVxServer
    
    Backend configurations for PTVx
    
    Configure PTV Service for used by AO
    
    PTV web service configs in AO servers/netapp
    
    AO GCP nightly env.
    
    AO GCP nightly env.
    
    Server as a service in GCP AO
- EazyTMS
  EazyTMS
- PTV
  PTV
- Web Apps
  Web Apps
Security
Security
- GCP Secret Manager
- ISECure / wscli
- Microsoft Defender rollout
- Statement of Applicability — Use of Cryptography
- Audits
  Audits
  - Audit logs review (internal)
  - Access Reviews (internal)
    
    Access Reviews (internal)
    
    run_access_google_db.yml: GCP Database Access Reviews (internal)
- Auth
  Auth
  - SSO
  - Yubikey
  - Auth0
    
    Auth0
    
    Auth0: Errors
    
    Auth0: Taksi Helsinki
    
    Auth0: how to fetch API key
    
    Auth0: How to generate API keys for customer org usage
    
    Changing AUTH0 user email
    
    Aleksandar superset tests
    
    Aleksandar superset tests
    
    app give access to login.
  - Devops OpenVPN
    
    Devops OpenVPN
    
    Create a Devops VPN identity / certificate
  - ORY
    
    ORY
    
    Ahola Digital Auth ORY
    
    Azure connection token
    
    tms-booking customers (dev/old) ORY
  - Teleport Zero trust VPN
    
    Teleport Zero trust VPN
    
    Teleport: Session recording
    
    VPN: 0. Admin Commands cli
    
    Teleport Zero trust VPN 3. Upgrade update
    
    Teleport Zero trust VPN 3. Upgrade update
    
    Cluster update
    
    Teleport Zero trust VPN sec
    
    Teleport Zero trust VPN sec
    
    SSO GitHub
    
    Teleport teleport ssh, awx
    
    Teleport teleport ssh, awx
    
    Teleport Zero trust VPN: teleport ssh, awx: testing1
    
    VPN 1. Create users and rules
    
    VPN 1. Create users and rules
    
    Rules
    
    VPN 2. Access
    
    VPN 2. Access
    
    Teleport Zero trust VPN: 2. Access: Applications: users
    
    VPN 2. Access Applications, API's
    
    VPN 2. Access Applications, API's
    
    Teleport Zero trust VPN: 2. Access: Applications: tsh cli
    
    VPN 2. Access Databases
    
    VPN 2. Access Databases
    
    VPN: 2. Access: Databases: GCP SQL
    
    VPN 2. Access Databases Postgres
    
    VPN 2. Access Databases Postgres
    
    VPN: 2. Access: Databases: Postgres: Database service
    
    VPN 2. Access Databases Postgres pgAdmin access
    
    VPN 2. Access Databases Postgres pgAdmin access
    
    VPN: 2. Access: Databases: Postgres: pgAdmin access: install tsh
    
    VPN 2. Access Machine
    
    VPN 2. Access Machine
    
    VPN: 2. Access: Machine: Ubuntu
    
    VPN 2. Access Machine Windows
    
    VPN 2. Access Machine Windows
    
    VPN: 2. Access: Machine: Windows: Install
    
    VPN 2. Access Machine ID
    
    VPN 2. Access Machine ID
    
    VPN: 2. Access: Machine ID ??: 1
- Certificates
  Certificates
  - Apple certificates
    
    Apple certificates
    
    Renew apple cert in Google Admin console
  - SSL certificates
    
    SSL certificates
    
    SSL Certificate list
    
    SSL certificate usage
    
    SSL certificate renewals
    
    SSL certificate renewals
    
    Renew Digicert Certificate
    
    SSL certificate renewals: *.aholadigital.com
    
    SSL certificate renewals: *.online.aholadigital.com
    
    SSL certificate renewals: Current process (2026 — Let's Encrypt + Vault, all wildcards)
    
    SSL certificate renewals: Meshmoon
    
    SSL certificate renewals: aholatransport.com
    
    SSL certificate renewals: atspecial.fi
    
    SSL certificate renewals: online2.attracs.com
    
    SSL certificate renewals attracs.com
    
    SSL certificate renewals attracs.com
    
    SSL certificate renewals: Updating certificates attracs/ahola in Hetzner
- Cloud Run ClamAV
  Cloud Run ClamAV
- Dependency Track
  Dependency Track
- Firewall
  Firewall
  - Azure Firewall (IT)
    
    Azure Firewall (IT)
  - Cloudflare
    
    Cloudflare
  - Fortigate
    
    Fortigate
    
    Fortinet VPN
  - GCP Cloud Armor
    
    GCP Cloud Armor
    
    GCP Cloud Armor Geoblocking rules
    
    GCP Cloud Armor JSON parsing
    
    GCP Cloud Armor preconfigured WAF rule 8 kb inspection limit
  - GCP Firewall
    
    GCP Firewall
  - Hetzner Firewall
    
    Hetzner Firewall
  - SendGrid
    
    SendGrid
    
    How to create API keys in Sendgrid
    
    Sendgrid keys
    
    Create inbound parse and security
    
    Create inbound parse and security
    
    Short summary for creation
- Operating Procedures
  Operating Procedures
  - External Key Management
  - YubiKey hardware authentication
- SIEM
  SIEM
  - SecurityOnion
    
    SecurityOnion
    
    Admin
    
    Installing
    
    Monitoring
  - Wazuh
    
    Wazuh
    
    Auditor explain
    
    Rules
    
    Wazuh API: (delete old agent)
    
    Wazuh: Requirements
    
    Wazuh: agent installation (workstations)
    
    Wazuh: install
    
    Wazuh: settings
    
    config file
    
    Troubleshooting
    
    Troubleshooting
    
    steps
    
    Wazuh GCP integration
    
    Wazuh GCP integration
    
    GCP Logs Router sink
    
    Wazuh security
    
    Wazuh security
    
    Wazuh: security: Install add new Agent with password key
- Vault (Hashicorp)
  Vault (Hashicorp)
  - GCP Vault (old vault)
  - Hetzner Vault servers
  - Set up old vault connection
  - Vault (Hashicorp): Whitelisting new instances
  - Vault Hetzner GCP
  - commands
  - Hetzner Vault (new vault)
    
    Hetzner Vault (new vault)
    
    -vaultHetzner

Table of contents

Documents

Failed Update

Documents

GCP Patch Job Failure (TMS Server)
GCP Patch Job Failure on Production Server
post-update checks

Made with Material for MkDocs