CI Pipeline Optimizer

Slow CI pipelines are one of the most common productivity killers in software teams. A test suite that takes 30 minutes to run means developers context-switch away from their work, batch multiple changes into fewer PRs to avoid waiting, and eventually start skipping CI altogether. This skill addresses CI pipeline performance through four complementary strategies: intelligent test splitting across parallel workers, selective test execution based on code changes, aggressive caching of dependencies and build artifacts, and pipeline architecture that minimizes total wall-clock time. The techniques apply to any CI system but use GitHub Actions as the primary example, with patterns that transfer to GitLab CI, CircleCI, Jenkins, and other platforms.

Core Principles

1. Wall-Clock Time Is the Only Metric That Matters

CPU time, billable minutes, and total test count are secondary metrics. The developer waits for wall-clock time. A pipeline that uses 60 CPU-minutes across 10 parallel workers in 6 minutes is vastly preferable to a pipeline that uses 20 CPU-minutes sequentially in 20 minutes. Optimize for the duration the developer experiences.

2. Never Run Tests You Do Not Need

If a PR changes only documentation files, running the full test suite is waste. If a PR modifies only the frontend, running backend integration tests is waste. Selective test execution identifies the minimum set of tests needed to validate a specific change, with a safety net that defaults to running everything when the analysis is uncertain.

3. Cache Everything That Does Not Change

Dependencies, build artifacts, Docker layers, and browser binaries are the same across most CI runs. Downloading and building them from scratch on every run is unnecessary. Aggressive caching can eliminate minutes from every pipeline execution.

4. Split Tests by Duration, Not by File Count

Naive test splitting distributes files evenly across workers. But if one file contains a 5-minute test and another contains a 5-second test, the distribution is severely unbalanced. Intelligent splitting uses historical duration data to distribute tests so that all workers finish at approximately the same time.

5. Fail Fast, Report Completely

Run the fastest tests first. If unit tests catch a bug in 30 seconds, there is no reason to wait 10 minutes for e2e tests to also catch it. Structure the pipeline so the cheapest checks (lint, type-check) run first to provide the fastest possible feedback, but still collect complete results from all shards for thorough reporting.

Project Structure

ci-config/
  scripts/
    split-tests.ts
    detect-changes.ts
    cache-manager.ts
    timing-collector.ts
    pipeline-analyzer.ts
  config/
    test-groups.json
    change-map.json
    cache-config.json
  .github/
    workflows/
      ci-optimized.yml
      ci-selective.yml
      cache-warmup.yml
  package.json
  tsconfig.json

The scripts/ directory contains automation tools for test splitting, change detection, cache management, and pipeline analysis. The config/ directory holds the mapping from file changes to test groups, cache key definitions, and test group specifications.

Test Splitting Strategies

Duration-Based Test Splitting

Split tests across parallel workers based on historical execution times so that each worker runs for approximately the same duration:

// scripts/split-tests.ts

import { readFileSync, writeFileSync, existsSync } from 'fs';
import { execSync } from 'child_process';

interface TestTiming {
  file: string;
  duration: number;
  lastRun: string;
}

interface SplitResult {
  shardIndex: number;
  files: string[];
  estimatedDuration: number;
}

export function splitTestsByDuration(
  testFiles: string[],
  shardCount: number,
  timingsFile: string = 'test-timings.json'
): SplitResult[] {
  // Load historical timings
  const timings = loadTimings(timingsFile);

  // Create a list of files with their estimated duration
  const fileTimings: Array<{ file: string; duration: number }> = testFiles.map((file) => ({
    file,
    duration: timings.get(file) || estimateDefaultDuration(file),
  }));

  // Sort by duration descending (greedy algorithm: assign largest jobs first)
  fileTimings.sort((a, b) => b.duration - a.duration);

  // Initialize shards
  const shards: SplitResult[] = Array.from({ length: shardCount }, (_, i) => ({
    shardIndex: i,
    files: [],
    estimatedDuration: 0,
  }));

  // Greedy assignment: always assign to the shard with the least total duration
  for (const fileTiming of fileTimings) {
    const lightest = shards.reduce((min, shard) =>
      shard.estimatedDuration < min.estimatedDuration ? shard : min
    );
    lightest.files.push(fileTiming.file);
    lightest.estimatedDuration += fileTiming.duration;
  }

  return shards;
}

function loadTimings(timingsFile: string): Map<string, number> {
  const timings = new Map<string, number>();

  if (!existsSync(timingsFile)) {
    return timings;
  }

  try {
    const data: TestTiming[] = JSON.parse(readFileSync(timingsFile, 'utf-8'));
    for (const entry of data) {
      timings.set(entry.file, entry.duration);
    }
  } catch {
    console.warn(`Failed to parse timings file: ${timingsFile}`);
  }

  return timings;
}

function estimateDefaultDuration(file: string): number {
  // Heuristic estimates based on test type when no historical data exists
  if (file.includes('.e2e.') || file.includes('e2e/')) return 30000;
  if (file.includes('.integration.') || file.includes('integration/')) return 10000;
  if (file.includes('.spec.')) return 5000;
  return 3000; // Default estimate for unit tests
}

// CLI entry point for use in GitHub Actions
function main(): void {
  const shardIndex = parseInt(process.env.SHARD_INDEX || '0', 10);
  const shardCount = parseInt(process.env.SHARD_COUNT || '1', 10);

  // Discover all test files
  const testFilesOutput = execSync(
    'find . -name "*.test.ts" -o -name "*.spec.ts" | grep -v node_modules',
    { encoding: 'utf-8' }
  );
  const testFiles = testFilesOutput.trim().split('\n').filter(Boolean);

  const shards = splitTestsByDuration(testFiles, shardCount);
  const myShard = shards[shardIndex];

  if (!myShard) {
    console.error(`Invalid shard index ${shardIndex} for ${shardCount} shards`);
    process.exit(1);
  }

  console.log(
    `Shard ${shardIndex + 1}/${shardCount}: ${myShard.files.length} files, ~${(myShard.estimatedDuration / 1000).toFixed(1)}s`
  );

  // Write shard files list for the test runner to consume
  writeFileSync('shard-files.txt', myShard.files.join('\n'), 'utf-8');
}

main();

Collecting Test Timings

After each CI run, collect and store test execution timings for future split optimization:

// scripts/timing-collector.ts

import { readFileSync, writeFileSync, existsSync } from 'fs';

interface JestResult {
  testResults: Array<{
    testFilePath: string;
    perfStats: {
      runtime: number;
    };
  }>;
}

interface PlaywrightResult {
  suites: Array<{
    file: string;
    specs: Array<{
      tests: Array<{
        results: Array<{
          duration: number;
        }>;
      }>;
    }>;
  }>;
}

interface TestTiming {
  file: string;
  duration: number;
  lastRun: string;
}

export function collectJestTimings(resultsFile: string): TestTiming[] {
  const results: JestResult = JSON.parse(readFileSync(resultsFile, 'utf-8'));

  return results.testResults.map((test) => ({
    file: test.testFilePath.replace(process.cwd() + '/', ''),
    duration: test.perfStats.runtime,
    lastRun: new Date().toISOString(),
  }));
}

export function collectPlaywrightTimings(resultsFile: string): TestTiming[] {
  const results: PlaywrightResult = JSON.parse(readFileSync(resultsFile, 'utf-8'));
  const timings: TestTiming[] = [];

  for (const suite of results.suites) {
    let totalDuration = 0;
    for (const spec of suite.specs) {
      for (const test of spec.tests) {
        for (const result of test.results) {
          totalDuration += result.duration;
        }
      }
    }

    timings.push({
      file: suite.file,
      duration: totalDuration,
      lastRun: new Date().toISOString(),
    });
  }

  return timings;
}

export function mergeTimings(
  existing: TestTiming[],
  latest: TestTiming[]
): TestTiming[] {
  const map = new Map<string, TestTiming>();

  for (const timing of existing) {
    map.set(timing.file, timing);
  }

  // Merge with exponential moving average to smooth out outliers
  for (const timing of latest) {
    const prev = map.get(timing.file);
    if (prev) {
      // EMA with alpha = 0.3 gives 70% weight to history, 30% to new data
      const smoothedDuration = prev.duration * 0.7 + timing.duration * 0.3;
      map.set(timing.file, {
        file: timing.file,
        duration: Math.round(smoothedDuration),
        lastRun: timing.lastRun,
      });
    } else {
      map.set(timing.file, timing);
    }
  }

  return Array.from(map.values());
}

function main(): void {
  const timingsFile = 'test-timings.json';
  const existing: TestTiming[] = existsSync(timingsFile)
    ? JSON.parse(readFileSync(timingsFile, 'utf-8'))
    : [];

  let latest: TestTiming[] = [];

  if (existsSync('jest-results.json')) {
    latest = collectJestTimings('jest-results.json');
  } else if (existsSync('playwright-results.json')) {
    latest = collectPlaywrightTimings('playwright-results.json');
  }

  if (latest.length > 0) {
    const merged = mergeTimings(existing, latest);
    writeFileSync(timingsFile, JSON.stringify(merged, null, 2), 'utf-8');
    console.log(`Updated timings for ${latest.length} test files (${merged.length} total)`);
  } else {
    console.warn('No test results found to collect timings from');
  }
}

main();

Selective Test Execution

Change Detection Engine

Detect which files changed and determine which test groups need to run:

// scripts/detect-changes.ts

import { execSync } from 'child_process';
import { readFileSync, existsSync } from 'fs';

interface ChangeMap {
  patterns: Array<{
    glob: string;
    testGroups: string[];
    description: string;
  }>;
  testGroups: Record<
    string,
    {
      command: string;
      files?: string[];
      description: string;
    }
  >;
}

interface DetectedChanges {
  changedFiles: string[];
  testGroupsToRun: Set<string>;
  skipReason?: string;
}

export function detectChanges(baseBranch: string = 'main'): DetectedChanges {
  let changedFiles: string[];

  try {
    const diffOutput = execSync(
      `git diff --name-only origin/${baseBranch}...HEAD`,
      { encoding: 'utf-8' }
    );
    changedFiles = diffOutput.trim().split('\n').filter(Boolean);
  } catch {
    const diffOutput = execSync('git diff --name-only HEAD~1', {
      encoding: 'utf-8',
    });
    changedFiles = diffOutput.trim().split('\n').filter(Boolean);
  }

  if (changedFiles.length === 0) {
    return {
      changedFiles: [],
      testGroupsToRun: new Set(),
      skipReason: 'No files changed',
    };
  }

  const changeMap = loadChangeMap();
  const testGroups = new Set<string>();

  for (const file of changedFiles) {
    for (const pattern of changeMap.patterns) {
      if (matchGlob(file, pattern.glob)) {
        for (const group of pattern.testGroups) {
          testGroups.add(group);
        }
      }
    }
  }

  // Safety net: if no patterns matched, run all tests
  if (testGroups.size === 0) {
    testGroups.add('all');
  }

  return {
    changedFiles,
    testGroupsToRun: testGroups,
  };
}

function loadChangeMap(): ChangeMap {
  const configPath = 'ci-config/config/change-map.json';
  if (!existsSync(configPath)) {
    return getDefaultChangeMap();
  }
  return JSON.parse(readFileSync(configPath, 'utf-8'));
}

function getDefaultChangeMap(): ChangeMap {
  return {
    patterns: [
      {
        glob: 'src/api/**',
        testGroups: ['unit', 'api-integration'],
        description: 'API source changes trigger unit and integration tests',
      },
      {
        glob: 'src/components/**',
        testGroups: ['unit', 'component'],
        description: 'UI component changes trigger unit and component tests',
      },
      {
        glob: 'src/pages/**',
        testGroups: ['unit', 'e2e'],
        description: 'Page-level changes trigger unit and E2E tests',
      },
      {
        glob: 'src/lib/**',
        testGroups: ['unit'],
        description: 'Library changes trigger unit tests',
      },
      {
        glob: 'src/db/**',
        testGroups: ['unit', 'api-integration', 'e2e'],
        description: 'Database changes trigger all test types',
      },
      {
        glob: 'package.json',
        testGroups: ['all'],
        description: 'Dependency changes require full test run',
      },
      {
        glob: '*.config.*',
        testGroups: ['all'],
        description: 'Config changes require full test run',
      },
      {
        glob: '**/*.md',
        testGroups: ['docs-only'],
        description: 'Documentation-only changes skip tests',
      },
      {
        glob: '.github/**',
        testGroups: ['ci-only'],
        description: 'CI configuration changes need validation',
      },
    ],
    testGroups: {
      all: { command: 'npm test', description: 'Full test suite' },
      unit: { command: 'npm run test:unit', description: 'Unit tests only' },
      'api-integration': {
        command: 'npm run test:integration',
        description: 'API integration tests',
      },
      component: { command: 'npm run test:components', description: 'Component tests' },
      e2e: { command: 'npx playwright test', description: 'End-to-end tests' },
      'docs-only': {
        command: 'echo "No tests needed for docs-only changes"',
        description: 'Skip tests',
      },
      'ci-only': {
        command: 'echo "CI config changed - validate workflow syntax only"',
        description: 'Validate CI config',
      },
    },
  };
}

function matchGlob(filePath: string, pattern: string): boolean {
  const regexPattern = pattern
    .replace(/\*\*/g, '<<<GLOBSTAR>>>')
    .replace(/\*/g, '[^/]*')
    .replace(/<<<GLOBSTAR>>>/g, '.*')
    .replace(/\?/g, '.');
  return new RegExp(`^${regexPattern}$`).test(filePath);
}

function main(): void {
  const baseBranch = process.env.BASE_BRANCH || 'main';
  const result = detectChanges(baseBranch);

  console.log(`Changed files: ${result.changedFiles.length}`);
  console.log(`Test groups to run: ${Array.from(result.testGroupsToRun).join(', ')}`);

  if (result.skipReason) {
    console.log(`Skip reason: ${result.skipReason}`);
  }

  // Output for GitHub Actions
  const groups = Array.from(result.testGroupsToRun);
  const shouldRun = groups.length > 0 && !groups.includes('docs-only');
  console.log(`::set-output name=test-groups::${JSON.stringify(groups)}`);
  console.log(`::set-output name=should-run-tests::${shouldRun}`);
}

main();

Caching Strategies

Multi-Layer Cache Configuration

// scripts/cache-manager.ts

import { readFileSync, existsSync } from 'fs';
import { createHash } from 'crypto';

interface CacheLayer {
  name: string;
  paths: string[];
  keyFiles: string[];
  fallbackKeys: string[];
  maxAge: number;
}

interface CacheConfig {
  layers: CacheLayer[];
}

export function generateCacheKeys(config: CacheConfig): Array<{
  name: string;
  key: string;
  restoreKeys: string[];
  paths: string[];
}> {
  const platform = process.env.RUNNER_OS || process.platform;

  return config.layers.map((layer) => {
    const fileHashes = layer.keyFiles
      .filter((f) => existsSync(f))
      .map((f) => hashFile(f))
      .join('-');

    const key = `${platform}-${layer.name}-${fileHashes}`;

    const restoreKeys = layer.fallbackKeys.map(
      (fallback) => `${platform}-${layer.name}-${fallback}`
    );

    return {
      name: layer.name,
      key,
      restoreKeys,
      paths: layer.paths,
    };
  });
}

function hashFile(filePath: string): string {
  const content = readFileSync(filePath);
  return createHash('sha256').update(content).digest('hex').substring(0, 16);
}

export const DEFAULT_CACHE_CONFIG: CacheConfig = {
  layers: [
    {
      name: 'node-modules',
      paths: ['node_modules', '~/.pnpm-store'],
      keyFiles: ['pnpm-lock.yaml', 'package.json'],
      fallbackKeys: [''],
      maxAge: 604800000, // 7 days
    },
    {
      name: 'playwright-browsers',
      paths: ['~/.cache/ms-playwright'],
      keyFiles: ['package.json'],
      fallbackKeys: [''],
      maxAge: 2592000000, // 30 days
    },
    {
      name: 'build-cache',
      paths: ['.next/cache', 'dist', '.turbo'],
      keyFiles: ['tsconfig.json', 'next.config.js'],
      fallbackKeys: [''],
      maxAge: 86400000, // 1 day
    },
    {
      name: 'test-timings',
      paths: ['test-timings.json'],
      keyFiles: [],
      fallbackKeys: [''],
      maxAge: 2592000000, // 30 days
    },
  ],
};

GitHub Actions Optimized Pipeline

Full Optimized CI Workflow

# .github/workflows/ci-optimized.yml
name: Optimized CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  # Step 1: Detect changes and determine what to test
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      backend: ${{ steps.changes.outputs.backend }}
      frontend: ${{ steps.changes.outputs.frontend }}
      config: ${{ steps.changes.outputs.config }}
      docs-only: ${{ steps.changes.outputs.docs }}
      shard-count: ${{ steps.shards.outputs.count }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Detect changed file categories
        id: changes
        uses: dorny/paths-filter@v3
        with:
          filters: |
            backend:
              - 'src/api/**'
              - 'src/db/**'
              - 'src/lib/**'
            frontend:
              - 'src/components/**'
              - 'src/pages/**'
              - 'src/styles/**'
            config:
              - 'package.json'
              - 'pnpm-lock.yaml'
              - '*.config.*'
            docs:
              - '**/*.md'
              - 'docs/**'

      - name: Determine optimal shard count
        id: shards
        run: |
          if [[ "${{ steps.changes.outputs.config }}" == "true" ]]; then
            echo "count=4" >> $GITHUB_OUTPUT
          elif [[ "${{ steps.changes.outputs.backend }}" == "true" && "${{ steps.changes.outputs.frontend }}" == "true" ]]; then
            echo "count=4" >> $GITHUB_OUTPUT
          elif [[ "${{ steps.changes.outputs.backend }}" == "true" || "${{ steps.changes.outputs.frontend }}" == "true" ]]; then
            echo "count=2" >> $GITHUB_OUTPUT
          else
            echo "count=1" >> $GITHUB_OUTPUT
          fi

  # Step 2: Lint and type-check (fastest feedback)
  lint:
    runs-on: ubuntu-latest
    needs: detect-changes
    if: needs.detect-changes.outputs.docs-only != 'true'
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v2
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'

      - run: pnpm install --frozen-lockfile

      - name: Run lint
        run: pnpm lint

      - name: Run type check
        run: pnpm tsc --noEmit

  # Step 3: Unit tests (fast, parallel shards)
  unit-tests:
    runs-on: ubuntu-latest
    needs: [detect-changes, lint]
    if: needs.detect-changes.outputs.docs-only != 'true'
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v2
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'

      - run: pnpm install --frozen-lockfile

      - name: Restore test timings
        uses: actions/cache@v4
        with:
          path: test-timings.json
          key: test-timings-${{ github.ref }}
          restore-keys: |
            test-timings-refs/heads/main
            test-timings-

      - name: Run unit tests (shard ${{ matrix.shard }}/2)
        run: |
          pnpm vitest run \
            --reporter=json \
            --outputFile=vitest-results.json \
            --shard=${{ matrix.shard }}/2

      - name: Collect test timings
        if: always()
        run: npx tsx ci-config/scripts/timing-collector.ts

      - name: Save test timings
        if: always()
        uses: actions/cache/save@v4
        with:
          path: test-timings.json
          key: test-timings-${{ github.ref }}-${{ github.run_id }}-shard-${{ matrix.shard }}

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: unit-results-shard-${{ matrix.shard }}
          path: vitest-results.json
          retention-days: 7

  # Step 4: Integration tests (medium speed, conditional)
  integration-tests:
    runs-on: ubuntu-latest
    needs: [detect-changes, lint]
    if: |
      needs.detect-changes.outputs.backend == 'true' ||
      needs.detect-changes.outputs.config == 'true'
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: test
          POSTGRES_DB: test
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v2
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'

      - run: pnpm install --frozen-lockfile

      - name: Run integration tests
        run: pnpm run test:integration
        env:
          DATABASE_URL: postgresql://postgres:test@localhost:5432/test

  # Step 5: E2E tests (slowest, most shards, conditional)
  e2e-tests:
    runs-on: ubuntu-latest
    needs: [detect-changes, lint]
    if: |
      needs.detect-changes.outputs.frontend == 'true' ||
      needs.detect-changes.outputs.config == 'true'
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v2
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'

      - run: pnpm install --frozen-lockfile

      - name: Cache Playwright browsers
        uses: actions/cache@v4
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium

      - name: Build application
        run: pnpm build

      - name: Run E2E tests (shard ${{ matrix.shard }}/4)
        run: npx playwright test --shard=${{ matrix.shard }}/4

      - name: Upload failure artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: e2e-results-shard-${{ matrix.shard }}
          path: test-results/
          retention-days: 7

Cache Warmup Workflow

Pre-populate caches on a schedule so the first PR of the day gets warm caches:

# .github/workflows/cache-warmup.yml
name: Cache Warmup

on:
  schedule:
    - cron: '0 6 * * 1-5'  # Weekdays at 6 AM UTC
  workflow_dispatch:

jobs:
  warmup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v2
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: pnpm install --frozen-lockfile