Cloud Run ClamAV
PoC investigation document draft: (TODO make this actual document) These two applications will have files uploaded to our bucket and should be scanned for malware. Need to figure out a way to do this.
e-learning-nightly bucket: https://console.cloud.google.com/storage/browser/e-learning-nightly;tab=objects?forceOnBucketsSortingFiltering=true&project=attracs-services-195009&prefix=&forceOnObjectsSortingFiltering=false
GCP documentation about the issue: https://cloud.google.com/architecture/automate-malware-scanning-for-documents-uploaded-to-cloud-storage
https://github.com/GoogleCloudPlatform/docker-clamav-malware-scanner
instructions to how install on cloud run: https://cloud.google.com/architecture/automate-malware-scanning-for-documents-uploaded-to-cloud-storage/deployment?hl=en
needed API's * Artifact Registry API * Cloud Run Admin API * Eventarc API * Cloud Logging API * Cloud Build API * Cloud Scheduler API * Cloud Pub/Sub API
Cloudrun did not work so had to build using linux GCP CLI:
REGION=europe-north1 LOCATION=eu PROJECT_ID=henri-test-project-123413 SERVICE_NAME="malware-scanner" SERVICE_ACCOUNT="malware-scanner@henri-test-project-123413.iam.gserviceaccount.com"
gsutil mb -l "eu" "gs://unscanned-henri-test-project-123413" gsutil mb -l "eu" "gs://quarantined-henri-test-project-123413" gsutil mb -l "eu" "gs://clean-henri-test-project-123413" gsutil mb -l "eu" "gs://cvd-mirror-henri-test-project-123413"
gcloud iam service-accounts create malware-scanner
gsutil iam ch \ "serviceAccount:malware-scanner@henri-test-project-123413.iam.gserviceaccount.com:objectAdmin" \ "gs://unscanned-henri-test-project-123413"
gsutil iam ch \ "serviceAccount:malware-scanner@henri-test-project-123413.iam.gserviceaccount.com:objectAdmin" \ "gs://clean-henri-test-project-123413"
gsutil iam ch \ "serviceAccount:malware-scanner@henri-test-project-123413.iam.gserviceaccount.com:objectAdmin" \ "gs://quarantined-henri-test-project-123413"
gsutil iam ch \ "serviceAccount:malware-scanner@henri-test-project-123413.iam.gserviceaccount.com:objectAdmin" \ "gs://cvd-mirror-henri-test-project-123413"
sed "s/-bucket-name/-henri-test-project-123413/" config.json.tmpl > config.json
python3 -m venv pyenv . pyenv/bin/activate pip3 install crcmod cvdupdate ./updateCvdMirror.sh "cvd-mirror-henri-test-project-123413" deactivate
gsutil ls "gs://cvd-mirror-henri-test-project-123413/cvds"
Needs to be run on correct folder which is: docker-clamav-malware-scanner/cloudrun-malware-scanner
gcloud beta run deploy "malware-scanner" \ --source . \ --region "europe-north1" \ --no-allow-unauthenticated \ --memory 4Gi \ --cpu 1 \ --concurrency 20 \ --min-instances 1 \ --max-instances 2 \ --no-cpu-throttling \ --cpu-boost \ --project="henri-test-project-123413" \ --service-account="malware-scanner@henri-test-project-123413.iam.gserviceaccount.com"
Service [malware-scanner] revision [malware-scanner-00001-2kk] has been deployed and is serving 100 percent of traffic. Service URL: https://malware-scanner-3fqkbxftga-lz.a.run.app
Testing with: curl -D - -H "Authorization: Bearer $(gcloud auth print-identity-token)" \ https://malware-scanner-3fqkbxftga-lz.a.run.app
If you're using an existing project that was created before April 8, 2021, add the iam.serviceAccountTokenCreator role to the Pub/Sub service account:
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)") PUBSUB_SERVICE_ACCOUNT="service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:${PUBSUB_SERVICE_ACCOUNT}"\ --role='roles/iam.serviceAccountTokenCreator'
STORAGE_SERVICE_ACCOUNT=$(gsutil kms serviceaccount -p "henri-test-project-123413") service-950403877430@gs-project-accounts.iam.gserviceaccount.com
gcloud projects add-iam-policy-binding "henri-test-project-123413" \ --member "serviceAccount:${STORAGE_SERVICE_ACCOUNT}" \ --role "roles/pubsub.publisher"
gcloud run services add-iam-policy-binding "${SERVICE_NAME}" \ --region="${REGION}" \ --member "serviceAccount:${SERVICE_ACCOUNT}" \ --role roles/run.invoker gcloud projects add-iam-policy-binding "${PROJECT_ID}" \ --member "serviceAccount:${SERVICE_ACCOUNT}" \ --role "roles/eventarc.eventReceiver"
BUCKET_NAME="unscanned-henri-test-project-123413" gcloud eventarc triggers create "trigger-unscanned-henri-test-project-123413-malware-scanner" \ --destination-run-service="malware-scanner" \ --destination-run-region="europe-north1" \ --location="eu" \ --event-filters="type=google.cloud.storage.object.v1.finalized" \ --event-filters="bucket=unscanned-henri-test-project-123413" \ --service-account="malware-scanner@henri-test-project-123413.iam.gserviceaccount.com"
SUBSCRIPTION_NAME=$(gcloud eventarc triggers describe \ "trigger-${BUCKET_NAME}-${SERVICE_NAME}" \ --location="${LOCATION}" \ --format="get(transport.pubsub.subscription)") gcloud pubsub subscriptions update "${SUBSCRIPTION_NAME}" --ack-deadline=120
TESTING PORTION!
gsutil cp test-file.txt "gs://unscanned-henri-test-project-123413"
gsutil ls -r "gs://clean-henri-test-project-123413"
gsutil ls -r "gs://unscanned-henri-test-project-123413"
echo -e 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' \ | gsutil cp - "gs://unscanned-henri-test-project-123413/eicar-infected.txt"
gsutil ls -r "gs://quarantined-henri-test-project-123413"
gcloud scheduler jobs run "malware-scanner-mirror-update" --location="europe-north1"
gcloud scheduler jobs create http \ "malware-scanner-mirror-update" \ --location="europe-west3" \ --project="henri-test-project-123413" \ --schedule="33 */2 * * *" \ --oidc-service-account-email="malware-scanner@henri-test-project-123413.iam.gserviceaccount.com" \ --uri="https://malware-scanner-3fqkbxftga-lz.a.run.app" \ --http-method=post \ --message-body='{"kind":"schedule#cvd_update"}' \ --headers="Content-Type=application/json"