Arq Backup Monitor using AWS Lambda, API Gateway, S3

November 13, 2018

Arq Backup is a great personal backup product that lets you backup your Mac to your cloud storage provider of choice . I use Arq Backup for all the users of the many computers we have in the house. It works great, is fairly cheap, and gives you a high level of control over

  • what gets backed up (folder selection, exclusion rules)
  • when it gets backed up (scheduling, pause during hours)
  • security of backups (client side encryption)

One challenge is to make sure backups are running successfully for each user on each computer. If a backup doesn't happen for 30 days I want to know about it.

I have constructed a two-part solution, Backup Marker and Lambda Monitor

Backup Marker

Write a marker meta-data file to AWS S3 after completion of each backup.

  • Create a new bucket to hold marker files arq-example-monitor
  • Create a service role for API Gateway that allows s3:PutObject into this bucket
  • Create an API Gateway service that integrates to S3 to upload the file
    • Use service role created above
    • Create an API Key, Usage Plan
    • Set the API Gateway method to require the API Key
    • Optionally set custom domain for the API
    • Deploy the API
  • Add the API Key to your Mac keychain
    • Replace zzz with API Key
    • /usr/bin/security add-generic-password -a arq-finish-api-key -s arq-finish-api-key -w zzz -U -T /usr/bin/security
  • Create shell script arq-finish.sh to upload the marker file to S3 via API Gateway
    • See code below, set the base URL of the API
    • Put somewhere like /Users/Shared
    • Give world execute, chmod 755 /Users/Shared/arq-finish.sh
  • Tell Arq to execute the shell script after each backup completes
    • Set Arq -> Preferences -> Destinations -> (Select) -> Edit -> Before and After Backup -> Run shell script after backup completes to /Users/Shared/arq-finish.sh
    • If one user has multiple backup destinations pass an additional argument to the script /Users/Shared/arq-finish.sh photo

arq-finish.sh

#!/bin/bash
BASE_URL=https://api.example.com/arq
API_KEY=`/usr/bin/security find-generic-password -a arq-finish-api-key -w`
ARG=${1:-main}
BACKUP_SET="`/bin/hostname -s`.`/usr/bin/whoami`.${ARG}"
FILE=${BACKUP_SET}.txt
URL=${BASE_URL}/${FILE}

/usr/bin/curl --header "x-api-key: $API_KEY" --upload-file - $URL << EOF
`/bin/date`
Arq `/usr/bin/defaults read /Applications/Arq.app/Contents/Info CFBundleVersion`
`/usr/bin/uname -a`  
EOF

Lamba Monitor

Periodically run a lambda function to check the age of the marker files and send an alert if too old.

  • Create SNS Topic ArqBackupMonitorAlert and subscribe your email address
  • Create a service role for Lambda which contains permission to s3:ListBucket and s3:GetObject on arq-example-monitor, and sns:Publish on ArqBackupMonitorAlert
  • Create ArqBackupMonitor Lambda
    • Python 3 runtime
    • Code below
    • Set variables for at least ARQ_BUCKET and ARQ_SNS_ARN
    • Create CloudWatch trigger to execute lambda every 10 days (set temporarily to every 5 minutes so you can test)

ArqBackupMonitor.py

import boto3
import os
from datetime import datetime, timezone, timedelta

def lambda_handler(event, context):
  arq_region = os.environ.get('AWS_REGION', 'us-east-1')
  arq_max_age_days = int(os.environ.get('ARQ_MAX_AGE_DAYS', '30'))
  arq_alert_subject = os.environ.get('ARQ_SNS_SUBJECT', 'Arq Monitor Alert')
  arq_bucket = os.environ.get('ARQ_BUCKET', 'arq-example-monitor')
  arq_alert_sns_arn = os.environ.get('ARQ_SNS_ARN', 
    'arn:aws:sns:us-east-1:12121212:ArqBackupMonitorAlert')
  
  too_old=''
  oldest_allowed = datetime.now(timezone.utc) - timedelta(days=arq_max_age_days)
  
  session = boto3.Session(region_name=arq_region)
  s3 = session.resource('s3')
  bucket = s3.Bucket(arq_bucket)
  for object in bucket.objects.all():
    if object.key.endswith('.txt'):
      if object.last_modified < oldest_allowed:
        too_old = too_old + object.key[:-4] + ' ' 
          + object.last_modified.strftime('%Y-%m-%d') + '\n'
 
  if len(too_old) > 0:
    msg = 'Arq backups are too old:\n\n' + too_old
    print(msg)
    sns = session.resource('sns')
    topic = sns.Topic(arq_alert_sns_arn)
    topic.publish(Subject=arq_alert_subject, Message=msg)
  
  return 'ArqBackupMonitor.Success'

Summary

  • After every successful Arq backup upload marker file via API Gateway to AWS S3
  • Every 10 days run a Lamba function that reads the marker files
  • If any marker file is older than 30 days send a notification to SNS Topic
  • SNS Topic sends notifcation via email

Note: Arq has a new product Arq Cloud Backup - full cloud backup solution including storage, like BackBlaze. I have installed Arq Cloud Backup up on my parents’ computers → it works great , “set it and forget it”. It also is incredibly fast, the Wasabi storage backend is speedy .