adi-devops-ci-instance-stable-1

Docker-cleanup cron job

This instance has the following cron job:

# Clean up ci-pipeline so we don't keep running out of disk space. -Miro 2025-06-03
# 1. Stop ci since it uses the volumes
# 2. List the volumes we want to delete and passs them to docker volume rm
# 3. Start the ci back up
# 4. Lastly write logs
0 14 * * 0 ( supervisorctl stop ci.aholadigital.com && docker volume ls -q | grep -E '^(eazytms-test_volume_|adi-ci-builder-shared)' | xargs -r docker volume rm && docker system prune -af && supervisorctl start ci.aholadigital.com ) >> /var/log/docker-cleanup.log 2>&1

The purpose of this job is to clean up any ci-bot build remains to keep the disk space free. stdout and err logs are forwarded to a log file.

Testing the cron job

# ( supervisorctl stop ci.aholadigital.com && docker volume ls -q | grep -E '^(eazytms-test_volume_|adi-ci-builder-shared)' | xargs -r docker volume rm && docker system prune -af && supervisorctl start ci.aholadigital.com ) >> /var/log/docker-cleanup.log 2>&1
root@adi-devops-ci-instance-stable-1:/etc/logrotate.d# cat /var/log/docker-cleanup.log 
ci.aholadigital.com: stopped
Deleted Images:
untagged: adi-ci-builder:local
deleted: sha256:2e6ebffabe0e928f36149b631941bb35d04c0efdcc74d98115153626767b06a2
deleted: sha256:f1417b94227b043eb3b1d7f0c5d80a59acd752dd8310429041ecf4c8b3c741fc
deleted: sha256:e1e574fb72d674b4f9ddd69e62e76b4b0175d34bf0c759760cccef32e3355bb3
...
...
deleted: sha256:c97e07a1d89a8a7f5871a963cb9ccd7c98a67224b95be63d77f2949c17c32344
untagged: testcontainers/ryuk:0.8.1
untagged: testcontainers/ryuk@sha256:bf3f74a47dee0acda89aba4b2fc9c7fdcf994a084db02a2d06566f07baae022e
...
untagged: ubuntu:24.04
untagged: ubuntu@sha256:b59d21599a2b151e23eea5f6602f4af4d7d31c4e236d22bf0b62b86d2e386b8f
deleted: sha256:bf16bdcff9c96b76a6d417bd8f0a3abe0e55c0ed9bdb3549e906834e2592fd5f
deleted: sha256:a8346d259389bc6221b4f3c61bad4e48087c5b82308e8f53ce703cfc8333c7b3

Total reclaimed space: 3.656GB
ci.aholadigital.com: started
ci.aholadigital.com: stopped
adi-ci-builder-shared
eazytms-test_volume_abcd1234
Deleted Images:
untagged: ubuntu:24.04
untagged: ubuntu@sha256:b59d21599a2b151e23eea5f6602f4af4d7d31c4e236d22bf0b62b86d2e386b8f
untagged: adi-ci-builder:local
deleted: sha256:bb408c82b360c7eb43f70a1ebb377e75ac25bbb56cf0c332283836f726d4253e
deleted: sha256:b94468effdda0e7a6f0af798376a06c69948090466ed01c7131ce927877f6168
deleted: sha256:cd659f3501dda8b2a46d72ceede863e58c47d7b9e2cf56630b63e6c4ee0ea1c7
...
...
untagged: testcontainers/ryuk:0.8.1
untagged: testcontainers/ryuk@sha256:bf3f74a47dee0acda89aba4b2fc9c7fdcf994a084db02a2d06566f07baae022e
...
deleted: sha256:db101830ef8975cc56d7ccafc14f9735a87b05f39002d42ea43e097523fda808
deleted: sha256:af9a70194aa4d12f967dbd4bcb1ce9c98ba42adb4ec05536080fd4560155e809

Total reclaimed space: 3.656GB
ci.aholadigital.com: started

Docker-cleanup log-rotate

/etc/logrotate.d/docker-cleanup

/var/log/docker-cleanup.log {
    su root root
    weekly
    rotate 4
    compress
    missingok
    notifempty
    copytruncate
}

su root root ìs there because of this error:

error: skipping "/var/log/docker-cleanup.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Testing log-rotate

# logrotate --debug /etc/logrotate.d/docker-cleanup
warning: logrotate in debug mode does nothing except printing debug messages!  Consider using verbose mode (-v) instead if this is not what you want.

reading config file /etc/logrotate.d/docker-cleanup
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state

Handling 1 logs

rotating pattern: /var/log/docker-cleanup.log  weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/docker-cleanup.log
Creating new state
  Now: 2025-06-03 09:55
  Last rotated at 2025-06-03 09:00
  log does not need rotating (log has already been rotated)

Manual cleanup

gcloud compute ssh adi-devops-ci-instance-stable-1 --project=attracs-devops --tunnel-through-iap
...

# stop ci.aholadigital.com process in managevisor

sudo bash
docker system prune -a

docker volume ls | grep "ci"
# remove all adi ci volumes, you can make this a single line with xargs or something, i have not bothered
docker volume rm adi-ci-builder-shared adi-ci-builder-temp_adi-eazytms_9bfd9047

# start ci.aholadigital.com process in managevisor