Skip to content

Grafana migration from old Hetzner to Hetzner Cloud

0. Grafana to hetzner steps

Replika of the old-grafana. - done Create duplicate channels for alerts. -done

follow alerts in grafana, if the alerts form old grafana are == to the new grafana hetzner. Remove old alert and switch _hetzner alert to the normal channel.

1. folders and their panels with dashboards

  • General

    • Espoo statistics (dashboards 0 or error need check)
      • Espoo customers with tokens
        • No data need to check.
        • Error: Datasource Attracs Google Cloud PostgreSQL - tms_booking was not found
      • Kulkukeskus manage users with tokens (updated hourly)
        • Data source: MySQL SELECT updated AS "time", espoo_users_login_countcol FROM grafana.espoo_users_login_count ORDER BY updated
      • Kulkukeskus rivers with tokens
        • espoo_drivers_with_token
        • espoo_drivers_total
    • PostgreSQL Database
      • General Counters, CPU, Memory and File Descriptor Stats (9 panels)
        1. Version
        2. Start Time
        3. Average CPU Usage
        4. Current fetch data
        5. Current insert data
        6. Average Memory Usage
        7. Current update data
        8. Max Connections
        9. Open File Descritors
      • Settings (7)
        1. Shared Buffers
        2. Effective Cache
        3. Maintenance Work Mem
        4. Work Mem
        5. Max WAL Size
        6. Random Page Cost
        7. Max Worker Processes
      • Database Stats (18 panels)
        1. Active sessions
        2. Transactions
        3. pg_stat_database_tup_updated
        4. pg_stat_database_tup_fetched
        5. pg_stat_database_tup_inserted
        6. Lock tables
        7. pg_stat_database_tup_returned
        8. Idle sessions
        9. pg_stat_database_tup_deleted
        10. Cache Hit Rate
        11. Buffers (bgwriter)
        12. Conflicts/Deadlocks
        13. Temp File (Bytes)
        14. Checkpoint Stats
        15. Database size (Bytes))
        16. Top 6 Database size increase in 7 days (b)
        17. Top6 Database size increase in 1 day
        18. Top6 Database size increase in 1 day (b) alert
    • PostGressSQL
      • Active connections
        • datname; username; state; count_user

  • Ahola-AO

    • Log digest - clients
      • LOG-ENTRY: OCLPERF DAILY
      • LOG-ENTRY: EXCEPTIONS DAILY
      • LOG-ENTRY: OSS: BD Optimistic Locking Fail event DAILY
      • LOG ENTRY: OSS: DB Optimistic Locking Fail event
      • LOG-ENTRY: EXCEPTION
      • LOG-ENTRY: OCLPERF
      • LOG-ENTRIES FOR USER LOGS: beginArrivalAsLDT grouped by server
        • no data
      • LOG-ENTRIES FOR USERS LOGS: beginArrivalsAsLDT grouped by ID
        • no data
      • LOG-ENTRIES FOR USER LOGS: User Logger off

  • applications

    • DriversAppServer

      • Sessions
      • Total responses by HTTP code per min

    • TMS (10 sections)

      • HTTP API (3 panels)

        1. HTTP API
        2. stable | Responce latency
        3. stable | Response latency (potentiall big latencies)

      • tms-drivers (3 panels)

        1. Vehicles which have update location within time period

          1. last_10sec
          2. last_60sec
          3. last_10min
          4. last_1h
          5. last_24
          6. total
        2. Drivers App Versions
        3. stable | Active WebSocket Clients

      • tms-planner (2 panels)

        1. Planner offer automation
        2. Combination suggester

      • Database (3 panels)

        1. PostgreSQL Connections
        2. Idle sessions
        3. Deadlock (alert on)
          1. alert message: TMS deadlock detected, please check why

      • RPC / Pubsub (4 panels)

        1. stable | RPC calls
        2. PubSub
        3. ExecuteBackoffs (alert on)
          1. no data need check!
          2. alert message : ExecuteBackOffs logged, please check!
        4. Oldest unacked message (alert on)
          1. alert message: Oldest unacked message age is over limit, please check.

      • WebSicket (3 panels)

        1. stable | WebSocket Events
        2. stable | Active WebSocket Clients
        3. stable | Auth

      • Server stats (6 panels)

        1. CPU Usage
        2. goroutines
        3. Network Traffic
        4. Process memory usage
        5. Server Status:
          1. tms-planner
          2. tms-booking
          3. tms-brivers
          4. tms-fleet
        6. Server free memory

      • App versions (1 panel)

        1. Service Versions
          1. build date; deploy; hash; instance; name; type; version

      • PubSub Detailed (10 panels)

        1. tms-backend-3.tms-booking (alert on)
          1. alert message when triggered: "Acked messages on pubsub subscriptions is suspiciously low, please check".
        2. tms-backend-4.tms-booking (alert on)
        3. tms-backend-3.tms-drivers (alert on)
        4. tms-backend-4.tms-drivers (alert on)
        5. tms-backend-3.tms-feedback (alert on)
        6. tms-backend-4-feedback (alert on)
        7. tms-backend-3.tms-fleet
        8. tms-backend-4.tms-fleet
        9. tms-backend-3.tms-planner
        10. tms-backend-4.tms-planner

      • Errors (3 panels)

        1. TMS Errors
        2. TMS Errors shortened
        3. TMS Errors shortened, last 7 days

    • Vehcodataservice

      1. Hits per endpoint per minute
      2. Average response times
      3. RPC calls
      4. PubSub
      5. PubSub Stable for Alarm (alert on)
        1. alert message: Vehco.stable.event pubsub channel has unusually low message count. Please check if we're receiving Vehcodata with vehcodataservice.
      6. Total responses by HTTP code per min (HAproxy is not aware of traffic duplication to nightly)
      7. Current total of incoming / outgoing bytes per min (HAproxy is not aware of traffic duplication to nightly)
      8. Number of transactions
        1. no data need to check
        2. I change this and it work
          1. Service was empty I select Monitoring and Metric name was empty select Time series count.
      9. PostgreSQL Connections
        1. No data need to check
      10. PostgreSQL Slow Queries - vehcodataservice
        1. No data need to check

    • aholapartnerportal

      1. Sessions
        1. no data need check
      2. Aholapartnerportal http requests per code
        1. no data need check

    • go metrics (on top we cam select deploy: nightly / stable and services: tms-fleet; tms-booking; tms-planner; tms-drivers

      1. GC spent time
      2. Next GC
      3. Memory in stack
      4. Total Used Memory
      5. Number of Live Objects
      6. Rate of Objects Allocated
      7. Rate of a Pointer Dereferences
      8. Gorutines
      9. Memory in off-Heap
      10. Memory in Heap
      11. Rates of Allocation
      12. GC duration quantile

    • tms_alerts

      here is metric for all tms services

      1. goroutines (alert on)

    • tower.attracs.com NEED TO CHECK no data.

      1. Tickets
      2. WebSocket
      3. Attracs Online API Vehicles
      4. Attracs Online API
      5. Online API Errors
      6. PubSub
      7. Nightly | response time
      8. Nightly requests

  • Camel

    • Camel Additionals no data need check
      1. Cloud Storage Received byte
      2. Cloud Storage Total bytes (daily)
      3. FTP Logins
      4. FTP Uploads
      5. SFTP Logins
      6. SFTP Uploads
      7. Pubsub
      8. Pubsub - inacked messages

  • Core

    • auth.attracs.com

      stable

      nightly

      1. stats
      2. stats by application
      3. response time
      4. service Details

    • connect.attracs.com

      Stable no data

      nightly

      1. WebSocket
      2. PubSub
      3. nightly | response time
      4. nightly | requests
      5. nightly | calls
      6. Service Details

    • location.attracs.com

      Sable no data

      nightly

      1. calls
      2. update
      3. ignored
        1. no data
      4. Service Details

    • manage.attracs.com

      stable no data

      nightly

      1. response time
      2. requests
      3. calls
      4. Service Details

    • resource.attracs.com

      stable no data

      nightly

      1. database
      2. calls
      3. Service Details

    • storage.attracs.com

      stable no data need check

      nightly

      1. response time
      2. requests
      3. calls
      4. Service Details

  • External-AO

    • Cbookings - Azure / IIS No data need check

      1. Cbooking latency
      2. Cbooking latency totals for .*sync
      3. Cbooking latency totals for .*departments.*sync
      4. Cbooking latency totals for .*organizations.*sync
      5. Cbooking latency totals for .*orders.*
      6. Cbooking latency totalCbooking latency total without .*sync
      7. Cbooking hit counts

    • External AO services

      1. Backendsrv-1-Cramo Production
      2. Backendsrv-0 - Peura Production
      3. BackendSRV-1 - Thermotrans Production
      4. BackendSrv-0 Roadari Production
      5. BackendSrv-1- OSTP Production
      6. BackendSrv-0 (alert on)
        1. BACKENDSRV-0 Service is down
      7. BackendSrv-1 (alert on)
        1. BACKENDSRV-1 Service is down

    • Log Digest - clients

      1. Log-ENTRY: Optimistic Locking Daily
      2. Log-ENTRY: Optimistic Locking (alert on)
        1. no data
        2. alert message: External AO might be experiencing OSS issue, please check
      3. Log-ENTRY: Exception (alert on)
        1. no data
        2. alert message: EXCEPTION's logged are over limit!
      4. Log-ENTRY: OCLPERF (log on)
        1. no data
        2. alert message: Logged OCLPERFs are over limit
      5. Log-ENTRIES For Users Logs

    • Log Digest - servers

      1. Log-Entry: Exception (alert on)
        1. alert message: EXCEPTIONs logged are over limit!
      2. LOG-ENTRY: OCLPERF (alert on)
        1. alert message: Logged OCLPERFs are over limit
      3. Log-Entry: Warning
      4. Log-Entry: INFO
      5. Log-Entry:AMCupdated by regnumber
      6. Log-Entry:AMCupdated by logname

    • Logging stats

      1. Log bytes ingested
      2. Exported log bytes per sink
      3. Exported log bytes per sink daily

    • RDP

      1. Certificate expiration for online2.attracs.com (alert on)
      2. Certificate expiration for cramo.attracs.com (alert on)
      3. Request latency (alert on)
        1. alert message: Request latency alert
      4. RDSGATEWAY - Firewall - Allowed TCP connections per ip in 1 day
      5. RDSGATEWAY - Firwall - Allower TCP connections per ip (health checks excluded) (alert on)
        1. alert message: Connection counts per ip over limit.
      6. RDSGATEWAY - Microsoft-Windows-TerminalServices-Gateway/Operational
      7. RDSGATEWAY - Event 4625 - Fail reason (alert on)
        1. alert message: RDSGATEWAY - Event 4625 - Fail reason alert
      8. RDSGATEWAY - Event 4625 - Username
      9. RDSBROKER - Microsoft-Windows-TerminalServices-SessionBroker/Operational EVENT 801 Successfully processed the connection
      10. RDSBROKER - Microsoft-Windows-TerminalServices-SessionBroker/Operational EVENT 801 Successfully processed the connection

    • Services

      1. BACKENDSRV-1 - cramo PRODUCTION
      2. BACKENDSRV-1 - roadari PRODUCTION
        1. no data
      3. BACKENDSRV-1 - peura PRODUCTION
        1. no data
      4. BACKENDSRV-1 - ssab PRODUCTION
        1. no data
      5. BACKENDSRV-1 - Thermotrans PRODUCTION
      6. BACKENDSRV-1 - ostp PRODUCTION
      7. BACKENDSRV-0 - roadari PRODUCTION
      8. BACKENDSRV-0 - ostp PRODUCTION
        1. no data
      9. BACKEND-0- cramo PRODUCTION
        1. no data
      10. BACKENDSRV-0 - peura PRODUCTION
      11. BACKENDSRV-0 - ssab PRODUCTION
        1. no data
      12. BACKENDSRV-0 (alert on)
        1. alert message:BACKENDSRV-0 Service is down
      13. BACKENDSRV-1 (alert on)
        1. alert message: BACKENDSRV-1 Service is down

    • Windows Hosts (select server: ip)

      • ALL HOSTS CPU USAGE FOR ALARMS

        1. CPU load (alert on) potential error
          1. Failed to load resource: the server responded with a status of 405 () prometheus
          2. alert message: CPU load is over limit, please check!

      • $Selected server ip

        1. CPU load
        2. system_threads
        3. System exception duspatches
        4. Memory
        5. Network
        6. Hard drive load
        7. Hard disk free space
        8. Services by state

  • General Alerting

    • 404

  • Google Cloud Monitoring

    • API stats

      1. Consumed API - request count per service
      2. Consumed API - request count per service
      3. Consumed API - request per credential
      4. Consumed API - request count per response code

    • Backups

      1. Backup sorage usage (alert on)
        1. alert message: attracs-backups -project storage space has increased over set limit, please check
      2. CloudSQL uptime
        1. no date
        2. message alert:
          1. Testing of DB restore to test SQL instace has failed. This means after sql export has been made and import has been done to separate sql instance, the query that has been used for testing that database works and data is new has failed. This can due one of these issues:
          2. 1) backup has failed
          3. 2) import has failed
          4. 3) sql query that is used to verify db is incorrect
      3. Number of objects in attracs-sql-backups bucket
      4. Database restore test results (alert on)
        1. no data

    • BigQuery

      1. Project for select:
        1. adi-audit-logs
        2. attracs-paijatsote-stable
        3. attracs-atspecial-stable
        4. adi-camel-additionals-nightly
        5. pg-databases-stable
        6. pg-databases-nightly
        7. attracs-devops
        8. Attracs-backups
        9. attracs-kulkukeskus
        10. attracs-ahola
        11. attracs-powdertrans
        12. attracs-logistics
        13. attracs-billing
        14. Attracs Nightly Services
        15. Attracs Services
      2. Panels:
        1. Query Execution Times
        2. Stored bytes for dataset None
        3. Uploaded rows for dataset None
        4. Query stats for project attracs-ahola per principalEmail and IP
        5. Job stats for project attracs-ahola per principalEmail and IP

    • ClamAV

      1. ClamAV scans
        1. no data
        2. alert message: ClamAV infected files value is bigger than 0, please check!

    • Cloud PubSub - Attracs Services (this folder need to be check\ optimization to much dashboards in one folder.

      1. Publish requests (PROJECT)
      2. Publish message size (project)
      3. Publish message operations (project)
      4. Topic byte cost, topic:All
      5. Topic updates, topic: All
      6. Topic updates, topic: All, grouped by operation type and response
      7. Ack latency, subscription: All
      8. Ack message count subscription: All
      9. Backlog size, subscription: All
      10. Subscription byte cost, subscription: All
      11. Subscription config updates: subscription: All
      12. Expired ack deadlines count: subscription: All
      13. Mod ack deadline message count: subscription: All
      14. ModifyAckDeadline message operations: subscription: all
      15. ModifAckDeadline requests: subscription:All
      16. Unacked messages: subscription: all
      17. Oldest unacked message age, subscription: all
      18. Acknowledge message operations, subscription: All
      19. Acknowledge message requests, subscription: All
      20. Sent message count, subscription: All
      21. Streamingpull Acknowledge requests, subscription:All
      22. Steamingpull message operations, subscription: All
      23. Streamingpull ModifyAckDeadline requests, subscription: All
      24. Streamingpull responses, subscription: All
      25. StreamingPull responses, subscription:All, grouped by result

    • Cloud PubSub - TMS

      1. Send message count (TMS-SERVER)
      2. Unacked message (TMS-server)

      3. Totals (7 panels)

        1. Sent Message count (TMS-SERVER)
        2. Unacked messages (TMS-SERVER)
        3. Publish requests (PROJECT)
        4. Publish message size (project)
        5. Publish message operations (project)
        6. Backlog Size (TMS-SERVER)
        7. ModifyAckDeadline requests (TMS-SERVER)
        8. Topic byte cost, topic: ALL
        9. Topic updates, topic: all

      4. Subscriptions (21 panels)

        1. Topic updates, topic: $Topic, grouped by operation type and response
        2. Ack latency, subscription: All
        3. Ack message count subscription: All
        4. Backlog size, subscription:All
        5. Subscription byte cost, subscription: All
        6. Subscription config updates: subscription: All
        7. Expired ack deadlines count: subscription: All
        8. Mod ack deadline

    • CloudSQL - MySQL

    • CloudSQL - Postgres - Connection Details
    • CloudSQL - postgres - Top Queries

    2. Alerts

  • Ahola-AO alert

    1. Set name
      1. Rule name: alert
    2. query and alert condition
      1. project: attracs-road-logging
      2. Service: loging
      3. Metric name: logging.google.com/user/aholaao_client_error_optimistic
    3. Alert evaluation behavior
      1. Folder: Ahola-AO
      2. Evaluation group: alert
      3. for: 1m
    4. Add details for your rule
      1. Alert ID:
      2. Dashboard UID:
      3. message:
    5. Notification
      1. Labels:
        1. alertname = alert
        2. rule_uid = jLb6vDG4z

  • Ahola-AO LOG-ENTRIES FOR USER LOGS: beginArrivalAsLDT grouped by ID alert

    • Dashboard UID :
    • Panel ID :
    • message
    • Labels :

  • applications Deadlocks alert

  • applications ExecuteBackoffs alert
  • applications goroutines alert
  • applications Oldest unacked message age, subscription:$Subscription alert
  • applications PubSub alert
  • applications tms-backend-3.tms-booking alert
  • applications tms-backend-3.tms-drivers alert
  • applications tms-backend-3.tms-feedback alert
  • applications tms-backend-4.tms-booking alert
  • applications tms-backend-4.tms-drivers alert
  • applications tms-backend-4.tms-feedback alert
  • applications Tower: Attracs Online REST API
  • External-AO BACKENDSRV-0 alert
  • External-AO BACKENDSRV-1 alert
  • External-AO Certificate expiration for cramo.attracs.com alert
  • External-AO Certificate expiration for online2.attracs.com alert
  • External-AO CPU load alert
  • External-AO LOG-ENTRY: EXCEPTION alert
  • External-AO LOG-ENTRY: EXCEPTION alert gLb6vvG4k
  • External-AO LOG-ENTRY: OCLPERF alert
  • External-AO LOG-ENTRY: OCLPERF alert zYxeDvM4z
  • External-AO LOG-ENTRY: OPTIMISTIC LOCKING alert
  • External-AO LOG-ENTRY: WARNING alert
  • External-AO RDSGATEWAY - Event 4625 - Fail reason alert
  • External-AO RDSGATEWAY - Firewall - Allowed TCP connections per ip (health checks excluded) alert
  • External-AO Request latency alert
  • General Alerting TOP6 Database size increase in 1 day (Bytes) alert
  • Google Cloud Monitoring /api/v1/booking/create latency alert
  • Google Cloud Monitoring backends alerts alert
  • Google Cloud Monitoring backends alerts alert 2YbeDDG4k
  • Google Cloud Monitoring Backup storage usage alert
  • Google Cloud Monitoring blocked requests alert
  • Google Cloud Monitoring Bytes used attracs-services alert
  • Google Cloud Monitoring CloudSQL Errors alert
  • Google Cloud Monitoring CPU utilization (ATTRACS LOGISTICS) alert
  • Google Cloud Monitoring CPU utilization (ATTRACS SERVICES) alert
  • Google Cloud Monitoring Database restore test results alert
  • Google Cloud Monitoring GCP
  • Google Cloud Monitoring GCP group
  • Google Cloud Monitoring Graph for transactions in idle_in_transaction_aborted_for_hour to get an alert alert
  • Google Cloud Monitoring Graph for transactions in idle_in_transaction_for_hour to get an alert alert
  • Google Cloud Monitoring HTTP REQUEST WITH 200 response alert
  • Google Cloud Monitoring Mem usage: attracs-core-backends-9qpt alert
  • Google Cloud Monitoring Mem usage: attracs-core-backends-multi-zone-nng8 alert
  • Google Cloud Monitoring Mem usage: attracs-logistics-backends-3cjz alert
  • Google Cloud Monitoring Mem usage: attracs-logistics-backends-vrr1 alert
  • Google Cloud Monitoring Mem usage: attracs-tms-backends-multi-zone-6ggh alert
  • Google Cloud Monitoring Mem usage: attracs-tms-backends-multi-zone-ckzf alert
  • Google Cloud Monitoring Oldest unacked message age, subscription:$Subscription alert
  • Google Cloud Monitoring PostgreSQL Deadlocks alert
  • Google Cloud Monitoring Security Updates alert
  • Google Cloud Monitoring Service status (ATTRACS LOGISTICS) alert
  • Google Cloud Monitoring Service status alert
  • Google Cloud Monitoring Spanner CPU usage alert
  • Google Cloud Monitoring Spanner storage used alert
  • Google Cloud Monitoring Unacked messages (TMS-SERVER) alert
  • Google Cloud Monitoring Unacked messages: tms subscriptions alert
  • Google Cloud Monitoring Uptime check - certificate alert
  • infra AholaAO
  • infra AO Server CPU 0%
  • infra Distribution daily stats alert
  • infra online-api-2.attracs.com - frontend responses by HTTP code alert
  • infra online-api-2.attracs.com - stable backend responses by HTTP code alert
  • infra online-api.attracs.com - Current number of sessions rate per second over last elapsed second alert
  • infra online-api.attracs.com - frontend responses by HTTP code alert
  • infra online-api.attracs.com - stable backend responses by HTTP code alert
  • infra Stable services: 5xx errors
  • infra Valhalla requests alert

3. Grafana OnCall

3.0 Grafana OnCall

[

github.com

https://github.com/grafana/oncall

](https://github.com/grafana/oncall)

We should investigate Grafana on-call since there is need to have a better alerting system that could help us to get real alerts to them who are working, giving the best effort to take actions quickly

  • We could centralize alerts to user(s) that are actually working daily, weekly with calendar integration?
    1. Yes, This is called on-call shift. We select users when they are on-call schedule. Also Users need to be mapped with slack user.
  • We no longer have to manage separate alerts from Grafana, Prometheus, etc. Everything is integrated and works with existing alerting sources and monitoring tools
    1. About this we again use existing alerts, but for notification we use hook from onCall.
    2. Using his different groups we can select with alerts are in what group. What alerts can wait or what alerts can't.
    3. also we have status for the alerts

  • allows us to actually sleep well at night knowing that only the relevant notifications will reach us.

    • Yes. For this we use escalation chains, when we set how notification is triggering. Like> after alert triggered > wait 5m > notifice user from on call schedule > wait >

  • What it costs (service/on our vm)

    • We will need to create new backend service for grafana onCall, and in grafana site to connect.
    • If we like to use some integration for SMS, Call we will need to implement some service like twilo?? but, slack and email notification and new one mobile app is okay.
    • The mobile app has some configs to be always online. It would be nice to have this in summer.

[

github.com

https://github.com/grafana/oncall#getting-started

](https://github.com/grafana/oncall#getting-started)

We use onCall plugin,but why we need grafana onCall backend service and connect with the plugin?

The Grafana OnCall plugin is designed to integrate with an incident management system, which typically requires a backend server to handle the communication and orchestration between Grafana and the incident management system. reasons why a backend server is necessary for the OnCall plugin: 1. Integration with incident management systems: The backend server acts as a bridge between Grafana and the incident management system. It handles the communication and data synchronization between the two systems, allowing Grafana to fetch and display information about ongoing incidents, on-call schedules, and other relevant data. 2. Authentication and authorization: The backend server manages user authentication and authorization for the OnCall plugin. It validates user credentials, ensures proper access control, and enforces security measures. This helps protect sensitive data and ensure that only authorized individuals can view and interact with incident-related information. 3. Webhook handling: Incident management systems often use webhooks to send real-time updates about incidents to external systems. The backend server can receive these webhook notifications and process them accordingly. It can trigger alerts, update Grafana dashboards, and initiate other actions based on the incoming data. 4. Data transformation and aggregation: The backend server can perform data transformation and aggregation tasks to provide a consolidated view of incidents within Grafana. It can normalize data from different incident management systems and present it in a standardized format, making it easier for users to analyze and monitor incidents from a single interface. 5. Customization and extensibility: With a backend server, you can extend the functionality of the OnCall plugin to suit your specific needs. You can implement custom workflows, integrate with additional systems, or apply business logic specific to your organization. The backend server acts as a flexible layer that allows you to tailor the plugin's behavior to align with your incident management processes. In summary, installing a backend server for the Grafana OnCall plugin enables seamless integration with incident management systems, provides security and access control, handles real-time updates, performs data transformation, and offers customization options. It enhances the functionality and usability of the plugin within the Grafana environment.

3.1 How to install

Grafana It's hosted on our DevOps.1 server which is in ahola on prem currently

For installing onCall we need: 1. 1. To instal OnCall engine this is backend for grafana. 2. And OnCal plugin installed on Grafana web configuration page. 3. and OnCall plugin to be connected with backend engine localhost:8080. This task will be easy, on our docker file we add: onCall docker extension sets. and on our grafana to install new plugin onCall. also in our grafana docker-compose file to add this environments environment: GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: grafana-oncall-app GF_INSTALL_PLUGINS: grafana-oncall-app

In grafana plugins find:

3.2 Contend

onCall: * * Alert groups

    \-In section Alert groups we can see all triggered alerts and their status with history of the escalation.

    • Users

      -here we can see created users in grafana. In this section we can see user configs for default or important notification. This is when in escalation chains we choose: "Start Default or Important notification for users:_____"

    • Integration

      in this scope we can create Integrations

      • This integrations are webhooks. We can create different webhooks for specific type of alerts. This integrations are paired with escalation chains. by default we go with Grafana Alerting

    • Escalation chains

      Here we can see\create escalation chain.

    • Schedules

      This section is for creating calendar with shifts.

    • Outgoing webhooks

      - Outgoing webhooks allow you to send alert details to a specified URL from Grafana OnCall. Once an outgoing webhook is configured, you can use it as a notification method.

      - For this example we can check Grafana: 3.3 Grafana OnCall: How to make integration local and cloud grafana notifications. (https://app.clickup.com/2496230/docs/2c5q6-9906/2c5q6-213541) using outgoing webhooks.

    • Maintenance

      -When we have problem wee can mute noisy sources or use for debugging and avoid bothering us.

    • Settings

3.3 How to make integration local and cloud grafana notifications

on my local env I created escalation chain where on the end I'm triggering webhook.

This webhook will be used for Calling the user. This webhook is created on free cloud grafana. Free plan on cloud can provide only 3 users. On my cloud grafana I created new Integration "Webhook" with escalation chain where I'm sending notification to the users.

because Escalation chain notification and Users notifications is different in my user configuration I set to be receive SMS and after 1 min to get call.

Simple demo of one escalation chain. 1. Alert is firing!!! 2. The configuration in Alerts & IRM we have set up webhook. where we will activate our escalation chain for notification.

Webhook is triggered and start the escalation. First post the alert in our chanel. and i tread is writhing the steps that will go. Start with Default notification for user admin. -This mean is triggering notifications for user admin.

Every user can create his own way of notification chain. There is 2 types Default and Important. and after that escalation is continuing... where we are triggering our webhook from cloudGrafana. For final result gloud grafana will call us, sms, email... (I don't find limitations for using onCall notifications. Only that free plan can support 3 users.) For this we need to have 2 apps integrated one from our grafana and second from cloud grafana.

3.4 Escalation Chain plan

3.4.0 Escalation Chain plan

I will create proper diagram for this algorithm diagram.

alarm
no need immediate action, most of the time is resolve by them self we can check tomorrow morning prepare to response for the alarm
firing firing firing
slack notification slack notification slack notification
#alert_0level #alert_1level #alert_devOps
wait 15min
Default notification onCall users.
Default notification ➝:
send slack and mobile App notification
wait 15.
/ if resolved =ok
else continue with email send
wait 5
if achnologic or resolve = ok
else continue with Next user.

3.4.1 Alerts prior

Nothing here yet

4. Public dashboards

The idea is we to create dashboard with Green / Red panels for our servers.

This example using Monitoring metric with Request latency. IF request latency is high like more than 10sec something is happening with this server so results will be hard availability response, or the server is down.

How to make dashboard public?

This is most easy step. In our docker of grafana ini file we need to add> GF_FEATURE_TOGGLES_ENABLE: publicDashboards

after restart we go to selected panel and in share menu we allow sharing to public.

Public dashboard * Every time we access the dashboard we make execution of queries * Depends the access * Here the problem will be more about GCP api access. If we use some metrics from GCP we make api access, so google will count every api access, the pay rate is little but if is happening some ddos this is no good. * but We run on permisess, so we will don't have much to worry about rapidly query execution.

some cautions limitations listed in grafana site:

Caution: Making your dashboard public could result in a large number of queries to the datasources used by our dashboard. This can be mitigated by utilizing the enterprise caching and/or rate limiting features.

Limitations

  • Panels that use frontend datasources will fail to fetch data.
  • Template variables are currently not supported, but are planned to be in the future.
  • Exemplars will be omitted from the panel.
  • Only annotations that query the -- Grafana -- datasource are supported.
  • Organization annotations are not supported.
  • Library panels are currently not supported, but are planned to be in the future.
  • Datasources using Reverse Proxy functionality are not supported.

What is Caching? Cached Queries are a way for you to build applications with charts and tables that load instantly. A common use case for caching is to prepare and pre-compute query results that are going to be embedded into live customer-facing dashboards.

Query caching

When query caching is enabled, Grafana temporarily stores the results of data source queries. When you or another user submit the exact same query again, the results will come back from the cache instead of from the data source (like Splunk or ServiceNow) itself. Note: Storing cached queries in-memory can increase Grafana’s memory footprint. In production environments, a Redis or Memcached backend is highly recommended.

Query caching benefits

We can create our Query caching, where we will fetch the data from the dashboards and we will store to our db and show that. and display on our website.

5. Blackbox exporter

  1. You need to clone the following repo:

[

github.com

https://github.com/prometheus/blackbox_exporter

](https://github.com/prometheus/blackbox_exporter)

Than you need to edit in blackbox.yml file the following code:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []  # Defaults to 2xx
      method: GET
  1. Than we create docker-compose file in the folder that we cloned the repo with the following code:
version: '3.2'
services:
  blackbox:
    image: prom/blackbox-exporter:latest
    ports:
    - 9115:9115
    command:
    - --config.file=/etc/blackbox/blackbox.yml
    volumes:
    - ./config/blackbox.yml:/etc/blackbox/blackbox.yml
  1. Then you make docker-compose up -d and you have blackbox exported docker created on port 9115.
  2. After that in the Prometheus configuration you need to add this code:
- job_name: blackbox
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
   - targets:
      - https://(aholadigital.com)/.      # This is example adress you need to change
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: localhost:9115   # Here you need to add the IP adress from Blackbox exporter

After that you go to Grafana ➝ Dashboard ➝ Import dashboard and you are importing the following dashboard:

[

grafana.com

https://grafana.com/grafana/dashboards/7587

](https://grafana.com/grafana/dashboards/7587)

1.