Module 07Automation~1 hour

Automating Repetitive Tasks

Stop typing the same commands every day. Build professional automation scripts that work identically locally and in CI.

Covers:MLO3MLO6

What is worth automating?

A useful rule of thumb from the DevOps community: if you do something more than twice, automate it. But not everything that is repeated is worth automating — automation has a cost too.

Task	Frequency	Automate?	Reason
Run the tests	Every change	Yes	Fast, repetitive, easy to script
Set up a new developer's machine	Monthly	Yes	Error-prone if manual, expensive to debug
Update a hardcoded value in a config file	Twice a year	Maybe	Simple enough to do manually
Generate quarterly financial report	4× per year	Yes	Complex, needs consistency
Name a feature	Occasionally	No	Requires human judgment

For DevOps workflows, the most valuable automation targets are: project setup, test running, code quality checks, dependency installation, and deployment steps. These are exactly what you automate in this module.

Anatomy of a production-grade script

Scripts for personal use can be loose. Scripts used by a team, or called by a CI pipeline, need more care:

bash — script template with best practices

#!/bin/bash
# set -e: exit immediately if any command fails
# set -u: treat unset variables as errors
# set -o pipefail: pipe fails if any command in it fails
set -euo pipefail

# Script metadata
SCRIPT_NAME="$(basename "$0")"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"

# Colour helpers
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
RESET='\033[0m'

log_info
()  { echo "${GREEN}[INFO]${RESET} $*"
; }
log_warn
()  { echo "${YELLOW}[WARN]${RESET} $*"
 >&2; }
log_error
() { echo "${RED}[ERROR]${RESET} $*"
 >&2; }

Script arguments and help text

A script that is easy to call and easy to understand is used correctly. Always include a usage function:

bash — usage and argument parsing

usage
() {
    cat <
Usage: $SCRIPT_NAME [OPTIONS] COMMAND

Commands:
  install    Install Python dependencies
  test       Run the test suite
  lint       Run flake8 style checks
  clean      Remove build artefacts
  all        Run install, lint, and test

Options:
  -h, --help   Show this help message
  -v, --verbose  Show verbose output
EOF
}

# Parse flags before the command
VERBOSE=false
while
 [[ "${1:-}" == -* ]]; do
    case
 "$1" in
        -h|--help)
 usage; exit 0 ;;
        -v|--verbose)
 VERBOSE=true; shift ;;
        *)
 log_error "Unknown option: $1"; usage; exit 1 ;;
    esac
done

Functions and reusability

Functions are the building blocks of maintainable scripts. Each function does one thing and returns an appropriate exit code:

bash — functions as tasks

install
() {
    log_info
 "Installing dependencies..."
    pip install -r requirements.txt -q
    log_info
 "Dependencies installed."
}

lint
() {
    log_info
 "Running linter..."
    if
 flake8 .; then
        log_info
 "Lint: passed"
    else
        log_error
 "Lint: failed"
        return 1
    fi
}

run_tests
() {
    log_info
 "Running tests..."
    if
 [ "$VERBOSE" = "true" ]; then
        pytest -v
    else
        pytest
    fi
}

clean
() {
    log_info
 "Cleaning..."
    rm -rf __pycache__ .pytest_cache .coverage htmlcov
    log_info
 "Cleaned."
}

Conditional automation

Scripts often need to behave differently in different contexts — locally vs in CI, staging vs production:

bash — environment detection

# Detect CI environment
if
 [ -n "${CI:-}" ]; then
    log_info
 "Running in CI mode"
    EXTRA_FLAGS="--tb=short"
else
    log_info
 "Running locally"
    EXTRA_FLAGS=""
fi

# GitHub Actions sets CI=true automatically
# Use this to adjust behaviour

run_tests
() {
    pytest $EXTRA_FLAGS
}

Git hooks: automating before commit

Git hooks are scripts that run at specific points in the Git workflow — before a commit, before a push, after a merge. They live in .git/hooks/.

The most useful hook is pre-commit — it runs automatically before git commit completes. If it exits with a non-zero code, the commit is aborted.

bash — .git/hooks/pre-commit

#!/bin/bash
set -euo pipefail

echo "Running pre-commit checks..."

# Only lint staged Python files
STAGED_PY="$(git diff --cached --name-only --diff-filter=ACM | grep \.py$)"

if
 [ -n "$STAGED_PY" ]; then
    flake8 $STAGED_PY
fi

echo "Pre-commit checks passed."

bash — installing the hook

# Make it executable
chmod +x .git/hooks/pre-commit

⚠ Hooks are not version-controlled by default

Files in .git/ are not committed to the repository, so hooks do not propagate to teammates automatically. Two solutions: (1) store the hook in a hooks/ directory in the repo and add a setup step that copies it; or (2) use the pre-commit framework (below), which handles this automatically.

Pre-commit framework

The pre-commit Python library manages hooks as configuration, making them shareable and version-controlled:

yaml — .pre-commit-config.yaml

repos:
  - repo: https://github.com/psf/black
    rev: 24.3.0
    hooks:
      - id: black

  - repo: https://github.com/pycqa/flake8
    rev: 7.0.0
    hooks:
      - id: flake8

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: check-yaml
      - id: end-of-file-fixer
      - id: detect-private-key

bash — installing pre-commit

pip install pre-commit

# Install the hooks into .git/hooks/
pre-commit install
pre-commit installed at .git/hooks/pre-commit

# Run manually on all files
pre-commit run --all-files

Connecting scripts to CI

Replace inline commands in your GitHub Actions workflow with calls to your script. This achieves pipeline parity — the pipeline does exactly what a developer does locally:

.github/workflows/ci.yml — using run.sh

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # Make the script executable (in case permissions were lost)
      - run: chmod +x run.sh
      - name: Install
        run: ./run.sh install
      - name: Lint
        run: ./run.sh lint
      - name: Test
        run: ./run.sh test

A complete automation example

Here is the complete run.sh drawing together everything from this module:

bash — run.sh (complete)

#!/bin/bash
set -euo pipefail

SCRIPT_NAME="$(basename "$0")"
VERBOSE=false

log
() { echo "[$SCRIPT_NAME] $*"
; }
err
() { echo "[$SCRIPT_NAME] ERROR: $*"
 >&2; exit 1; }

usage
() {
    echo "Usage: $SCRIPT_NAME {install|test|lint|clean|all}"
}

install
() {
    log
 "Installing dependencies"
    pip install -r requirements.txt -q
}

lint
() {
    log
 "Linting with flake8"
    flake8 .
}

run_tests
() {
    log
 "Running tests"
    pytest
}

clean
() {
    log
 "Cleaning artefacts"
    rm -rf __pycache__ .pytest_cache .coverage
}

# Route to the requested function
case
 "${1:-}" in
    install)
 install
 ;;
    test)
 run_tests
 ;;
    lint)
 lint
 ;;
    clean)
 clean
 ;;
    all)
 install
;
 lint
;
 run_tests
 ;;
    *)
 usage
; exit 1 ;;
esac

Key terms

set -euo pipefail

Combined safety flags: stop on error, treat unset vars as errors, propagate pipe failures.

function

A named, reusable block of shell commands.

exit code

0 = success, non-zero = failure. Controls whether CI passes or fails.

Git hook

A script in .git/hooks/ that runs at a specific point in the Git workflow.

pre-commit

A framework for managing shareable, version-controlled Git hooks.

pipeline parity

Local scripts and CI scripts are identical — no surprises between environments.

$CI

Environment variable set to true by most CI systems, including GitHub Actions.

case statement

A shell construct for routing to different functions based on an argument.

Exercises

✎ Lab exercises — approximately 55 minutes

Part A: Build run.sh

Write run.sh with four tasks: install, test, lint, clean.
Add a usage function that prints help when called with no arguments or --help.
Add set -euo pipefail. Deliberately make lint fail — does the script stop?
Add colour output (green for success, red for errors).

Part B: Git hooks

Write a pre-commit hook that runs flake8 on staged Python files.
Make it executable and test it by staging a file with a style error.
Fix the error. Verify the commit goes through.
Extension: add a check that prevents committing if any file contains the string TODO: REMOVE BEFORE MERGE.

Part C: CI integration

Update your GitHub Actions workflow from Module 6 to call run.sh instead of inline commands.
Push and verify the pipeline still passes.
Add ./run.sh all as a single step. Does the pipeline still report which step failed?
Consider: is it better to have one step calling all, or three separate steps? Why?

Part D: Pre-commit framework (extension)

Install the pre-commit library and create a .pre-commit-config.yaml.
Configure black for auto-formatting and flake8 for style checking.
Add a hook to detect accidentally committed private keys.
Run pre-commit run --all-files and fix any issues found.

← PreviousCI Pipelines in Practice Next →Virtualisation and Containers