Changelog Test Mapper

Every user-facing change in a software release should have corresponding test coverage that verifies the change works as intended. In practice, this mapping between changelog entries and test cases is rarely maintained, leading to releases where features ship without adequate verification and bug fixes lack regression tests. The Changelog Test Mapper bridges this gap by programmatically analyzing changelog entries, release notes, and commit histories, then mapping them to existing test cases and identifying coverage gaps. This skill covers the full pipeline: parsing changelogs in various formats, extracting change metadata, building a bidirectional mapping between changes and tests, scoring coverage completeness, and integrating the analysis into release workflows to prevent untested changes from reaching production.

Core Principles

1. Every Shipped Change Deserves a Test

The fundamental premise of changelog-test mapping is that if a change is important enough to appear in the changelog, it is important enough to have a test. Bug fixes need regression tests. New features need functional tests. Performance improvements need benchmark tests. Changes without tests are unverified promises.

2. Bidirectional Traceability

The mapping must work in both directions. Given a changelog entry, you should be able to find all related tests. Given a test, you should be able to find which changelog entries it covers. This bidirectional traceability enables both forward analysis ("is this release well-tested?") and backward analysis ("what does this test protect?").

3. Automation Over Manual Tracking

Manually maintaining a spreadsheet of change-to-test mappings does not scale. The mapping should be derived automatically from commit messages, PR descriptions, issue tracker links, and file-level change analysis. Manual annotations should supplement, not replace, automated discovery.

4. Coverage Scoring Is Contextual

Not all changelog entries need the same level of test coverage. A breaking API change requires more comprehensive testing than a documentation fix. The coverage scoring model must account for change severity, affected surface area, and historical defect rates.

5. Integrate at Release Time

The mapping analysis should run as part of the release process, not as an afterthought. A release checklist that includes "all changelog entries have mapped tests" prevents untested changes from shipping.

Project Structure

changelog-test-mapper/
  src/
    parsers/
      changelog-parser.ts
      conventional-commits-parser.ts
      github-releases-parser.ts
      keep-a-changelog-parser.ts
    analyzers/
      change-classifier.ts
      test-finder.ts
      coverage-mapper.ts
      gap-detector.ts
    mappers/
      file-change-mapper.ts
      semantic-mapper.ts
      annotation-mapper.ts
    reporters/
      coverage-reporter.ts
      gap-reporter.ts
      release-readiness-reporter.ts
    config/
      mapper-config.ts
    index.ts
  scripts/
    map-release.ts
    check-coverage.ts
    generate-report.ts
  tests/
    parsers/
    analyzers/
    mappers/
  package.json
  tsconfig.json

Changelog Parsing

Multi-Format Changelog Parser

Changelogs come in many formats. The parser supports Keep a Changelog, conventional commits, and GitHub release formats:

// src/parsers/changelog-parser.ts

export interface ChangelogEntry {
  id: string;
  type: 'added' | 'changed' | 'deprecated' | 'removed' | 'fixed' | 'security' | 'performance';
  description: string;
  version: string;
  date: string;
  scope?: string;
  breakingChange: boolean;
  issueRefs: string[];
  prRefs: string[];
  commitShas: string[];
  affectedFiles: string[];
  severity: 'critical' | 'major' | 'minor' | 'patch';
}

export interface ParsedChangelog {
  versions: Array<{
    version: string;
    date: string;
    entries: ChangelogEntry[];
  }>;
}

export function parseKeepAChangelog(content: string): ParsedChangelog {
  const versions: ParsedChangelog['versions'] = [];
  const versionRegex = /^## \[([^\]]+)\](?: - (\d{4}-\d{2}-\d{2}))?/gm;
  const typeRegex = /^### (Added|Changed|Deprecated|Removed|Fixed|Security)/gm;
  const entryRegex = /^- (.+)$/gm;

  const sections = content.split(/^## /gm).filter(Boolean);

  for (const section of sections) {
    const versionMatch = section.match(/^\[([^\]]+)\](?: - (\d{4}-\d{2}-\d{2}))?/);
    if (!versionMatch) continue;

    const version = versionMatch[1];
    const date = versionMatch[2] || '';
    const entries: ChangelogEntry[] = [];

    const typeSections = section.split(/^### /gm).filter(Boolean);

    for (const typeSection of typeSections.slice(1)) {
      const typeMatch = typeSection.match(/^(Added|Changed|Deprecated|Removed|Fixed|Security)/);
      if (!typeMatch) continue;

      const type = typeMatch[1].toLowerCase() as ChangelogEntry['type'];
      const lines = typeSection.split('\n').filter((l) => l.startsWith('- '));

      for (const line of lines) {
        const description = line.replace(/^- /, '').trim();
        const issueRefs = extractIssueRefs(description);
        const prRefs = extractPRRefs(description);
        const breakingChange = /breaking|BREAKING/.test(description);

        entries.push({
          id: generateEntryId(version, type, description),
          type,
          description,
          version,
          date,
          breakingChange,
          issueRefs,
          prRefs,
          commitShas: [],
          affectedFiles: [],
          severity: classifySeverity(type, breakingChange),
        });
      }
    }

    versions.push({ version, date, entries });
  }

  return { versions };
}

function extractIssueRefs(text: string): string[] {
  const matches = text.match(/#(\d+)/g) || [];
  return matches.map((m) => m.replace('#', ''));
}

function extractPRRefs(text: string): string[] {
  const matches = text.match(/\bPR[: ]?#?(\d+)/gi) || [];
  return matches.map((m) => m.replace(/\bPR[: ]?#?/i, ''));
}

function generateEntryId(version: string, type: string, description: string): string {
  const hash = description
    .toLowerCase()
    .replace(/[^a-z0-9]/g, '')
    .substring(0, 20);
  return `${version}-${type}-${hash}`;
}

function classifySeverity(
  type: ChangelogEntry['type'],
  breakingChange: boolean
): ChangelogEntry['severity'] {
  if (breakingChange) return 'critical';
  if (type === 'security') return 'critical';
  if (type === 'fixed') return 'major';
  if (type === 'removed' || type === 'deprecated') return 'major';
  if (type === 'changed') return 'minor';
  return 'patch';
}

Conventional Commits Parser

// src/parsers/conventional-commits-parser.ts

import type { ChangelogEntry } from './changelog-parser';

interface ConventionalCommit {
  sha: string;
  type: string;
  scope?: string;
  description: string;
  body?: string;
  breaking: boolean;
  issueRefs: string[];
  files: string[];
}

export function parseConventionalCommits(
  commits: ConventionalCommit[],
  version: string
): ChangelogEntry[] {
  const typeMapping: Record<string, ChangelogEntry['type']> = {
    feat: 'added',
    fix: 'fixed',
    perf: 'performance',
    security: 'security',
    refactor: 'changed',
    deprecate: 'deprecated',
  };

  return commits
    .filter((commit) => typeMapping[commit.type])
    .map((commit) => ({
      id: `${version}-${commit.sha.substring(0, 8)}`,
      type: typeMapping[commit.type] || 'changed',
      description: commit.description,
      version,
      date: new Date().toISOString().split('T')[0],
      scope: commit.scope,
      breakingChange: commit.breaking,
      issueRefs: commit.issueRefs,
      prRefs: [],
      commitShas: [commit.sha],
      affectedFiles: commit.files,
      severity: classifyCommitSeverity(commit),
    }));
}

function classifyCommitSeverity(commit: ConventionalCommit): ChangelogEntry['severity'] {
  if (commit.breaking) return 'critical';
  if (commit.type === 'security') return 'critical';
  if (commit.type === 'fix') return 'major';
  if (commit.type === 'feat') return 'minor';
  return 'patch';
}

Test Discovery and Mapping

File-Based Change-to-Test Mapper

The file-based mapper connects changelog entries to tests by analyzing which source files changed and finding tests that import or reference those files:

// src/mappers/file-change-mapper.ts

import { execSync } from 'child_process';
import { readFileSync, existsSync } from 'fs';
import { join, basename, dirname } from 'path';

interface TestMapping {
  changeEntryId: string;
  changeDescription: string;
  mappedTests: Array<{
    testFile: string;
    testName: string;
    confidence: 'high' | 'medium' | 'low';
    reason: string;
  }>;
  unmappedFiles: string[];
  coverageScore: number;
}

export function mapChangesToTests(
  affectedFiles: string[],
  testFiles: string[],
  changeEntryId: string,
  changeDescription: string
): TestMapping {
  const mappedTests: TestMapping['mappedTests'] = [];
  const mappedSourceFiles = new Set<string>();
  const unmappedFiles: string[] = [];

  for (const sourceFile of affectedFiles) {
    // Skip test files themselves
    if (isTestFile(sourceFile)) continue;

    let hasMapping = false;

    // Strategy 1: Co-located test files
    const colocatedTest = findColocatedTest(sourceFile, testFiles);
    if (colocatedTest) {
      mappedTests.push({
        testFile: colocatedTest,
        testName: `Tests for ${basename(sourceFile)}`,
        confidence: 'high',
        reason: 'Co-located test file',
      });
      mappedSourceFiles.add(sourceFile);
      hasMapping = true;
    }

    // Strategy 2: Convention-based test discovery
    const conventionTests = findConventionBasedTests(sourceFile, testFiles);
    for (const testFile of conventionTests) {
      if (!mappedTests.some((m) => m.testFile === testFile)) {
        mappedTests.push({
          testFile,
          testName: `Tests for ${basename(sourceFile)}`,
          confidence: 'medium',
          reason: 'Naming convention match',
        });
        mappedSourceFiles.add(sourceFile);
        hasMapping = true;
      }
    }

    // Strategy 3: Import analysis
    const importTests = findTestsByImport(sourceFile, testFiles);
    for (const testFile of importTests) {
      if (!mappedTests.some((m) => m.testFile === testFile)) {
        mappedTests.push({
          testFile,
          testName: `Imports ${basename(sourceFile)}`,
          confidence: 'medium',
          reason: 'Import dependency',
        });
        mappedSourceFiles.add(sourceFile);
        hasMapping = true;
      }
    }

    if (!hasMapping) {
      unmappedFiles.push(sourceFile);
    }
  }

  const totalSourceFiles = affectedFiles.filter((f) => !isTestFile(f)).length;
  const coverageScore =
    totalSourceFiles > 0 ? mappedSourceFiles.size / totalSourceFiles : 1;

  return {
    changeEntryId,
    changeDescription,
    mappedTests,
    unmappedFiles,
    coverageScore,
  };
}

function isTestFile(filePath: string): boolean {
  const name = basename(filePath);
  return (
    name.includes('.test.') ||
    name.includes('.spec.') ||
    name.includes('__tests__') ||
    filePath.includes('__tests__/')
  );
}

function findColocatedTest(sourceFile: string, testFiles: string[]): string | null {
  const dir = dirname(sourceFile);
  const name = basename(sourceFile).replace(/\.(ts|tsx|js|jsx)$/, '');

  const patterns = [
    join(dir, `${name}.test.ts`),
    join(dir, `${name}.test.tsx`),
    join(dir, `${name}.spec.ts`),
    join(dir, `${name}.spec.tsx`),
    join(dir, '__tests__', `${name}.test.ts`),
    join(dir, '__tests__', `${name}.test.tsx`),
  ];

  for (const pattern of patterns) {
    if (testFiles.includes(pattern)) {
      return pattern;
    }
  }

  return null;
}

function findConventionBasedTests(sourceFile: string, testFiles: string[]): string[] {
  const name = basename(sourceFile).replace(/\.(ts|tsx|js|jsx)$/, '');
  const matches: string[] = [];

  for (const testFile of testFiles) {
    const testName = basename(testFile).replace(/\.(test|spec)\.(ts|tsx|js|jsx)$/, '');
    if (testName === name) {
      matches.push(testFile);
    }
  }

  return matches;
}

function findTestsByImport(sourceFile: string, testFiles: string[]): string[] {
  const matches: string[] = [];
  const sourceBaseName = basename(sourceFile).replace(/\.(ts|tsx|js|jsx)$/, '');

  for (const testFile of testFiles) {
    if (!existsSync(testFile)) continue;

    try {
      const content = readFileSync(testFile, 'utf-8');
      if (
        content.includes(`from './${sourceBaseName}'`) ||
        content.includes(`from '../${sourceBaseName}'`) ||
        content.includes(`require('./${sourceBaseName}')`) ||
        content.includes(`/${sourceBaseName}'`)
      ) {
        matches.push(testFile);
      }
    } catch {
      // Skip files that cannot be read
    }
  }

  return matches;
}

Semantic Mapper

The semantic mapper uses keyword and pattern analysis to connect changelog descriptions to test names:

// src/mappers/semantic-mapper.ts

interface SemanticMatch {
  testFile: string;
  testName: string;
  confidence: number;
  matchedTerms: string[];
}

export function findSemanticMatches(
  changeDescription: string,
  testIndex: Array<{ file: string; name: string; description: string }>
): SemanticMatch[] {
  const changeTerms = extractKeyTerms(changeDescription);
  const matches: SemanticMatch[] = [];

  for (const test of testIndex) {
    const testTerms = extractKeyTerms(`${test.name} ${test.description}`);
    const matchedTerms = changeTerms.filter((term) =>
      testTerms.some(
        (testTerm) =>
          testTerm.includes(term) || term.includes(testTerm) || levenshteinSimilarity(term, testTerm) > 0.8
      )
    );

    if (matchedTerms.length > 0) {
      const confidence = matchedTerms.length / Math.max(changeTerms.length, 1);
      matches.push({
        testFile: test.file,
        testName: test.name,
        confidence: Math.min(1, confidence),
        matchedTerms,
      });
    }
  }

  return matches.sort((a, b) => b.confidence - a.confidence);
}

function extractKeyTerms(text: string): string[] {
  const stopWords = new Set([
    'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been', 'being',
    'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could',
    'should', 'may', 'might', 'can', 'shall', 'to', 'of', 'in', 'for',
    'on', 'with', 'at', 'by', 'from', 'as', 'into', 'through', 'during',
    'before', 'after', 'above', 'below', 'between', 'and', 'but', 'or',
    'not', 'no', 'nor', 'so', 'yet', 'both', 'each', 'all', 'any',
    'few', 'more', 'most', 'other', 'some', 'such', 'than', 'too',
    'very', 'just', 'about', 'when', 'where', 'how', 'what', 'which',
    'who', 'whom', 'this', 'that', 'these', 'those', 'it', 'its',
    'add', 'update', 'fix', 'remove', 'change', 'new', 'now',
  ]);

  return text
    .toLowerCase()
    .replace(/[^a-z0-9\s-]/g, ' ')
    .split(/[\s-]+/)
    .filter((word) => word.length > 2 && !stopWords.has(word));
}

function levenshteinSimilarity(a: string, b: string): number {
  const maxLen = Math.max(a.length, b.length);
  if (maxLen === 0) return 1;

  const matrix: number[][] = Array(a.length + 1)
    .fill(null)
    .map(() => Array(b.length + 1).fill(0));

  for (let i = 0; i <= a.length; i++) matrix[i][0] = i;
  for (let j = 0; j <= b.length; j++) matrix[0][j] = j;

  for (let i = 1; i <= a.length; i++) {
    for (let j = 1; j <= b.length; j++) {
      const cost = a[i - 1] === b[j - 1] ? 0 : 1;
      matrix[i][j] = Math.min(
        matrix[i - 1][j] + 1,
        matrix[i][j - 1] + 1,
        matrix[i - 1][j - 1] + cost
      );
    }
  }

  return 1 - matrix[a.length][b.length] / maxLen;
}

Gap Detection and Release Readiness

Gap Detector

// src/analyzers/gap-detector.ts

import type { ChangelogEntry } from '../parsers/changelog-parser';

interface CoverageGap {
  entry: ChangelogEntry;
  gapType: 'no-tests' | 'low-confidence' | 'partial-coverage' | 'missing-regression';
  riskLevel: 'critical' | 'high' | 'medium' | 'low';
  recommendation: string;
}

interface ReleaseReadiness {
  version: string;
  totalEntries: number;
  coveredEntries: number;
  coveragePercentage: number;
  gaps: CoverageGap[];
  readyForRelease: boolean;
  riskScore: number;
  blockers: string[];
  warnings: string[];
}

export function analyzeReleaseReadiness(
  entries: ChangelogEntry[],
  mappings: Map<string, { coverageScore: number; mappedTests: unknown[] }>,
  thresholds: { minCoverage: number; criticalRequireTests: boolean }
): ReleaseReadiness {
  const gaps: CoverageGap[] = [];
  const blockers: string[] = [];
  const warnings: string[] = [];
  let coveredEntries = 0;

  for (const entry of entries) {
    const mapping = mappings.get(entry.id);
    const coverageScore = mapping?.coverageScore || 0;
    const hasTests = mapping && mapping.mappedTests.length > 0;

    if (coverageScore >= 0.8) {
      coveredEntries++;
      continue;
    }

    let gapType: CoverageGap['gapType'];
    let recommendation: string;

    if (!hasTests) {
      gapType = 'no-tests';
      recommendation = `Add test coverage for: ${entry.description}`;
    } else if (coverageScore < 0.3) {
      gapType = 'low-confidence';
      recommendation = `Improve test mapping confidence for: ${entry.description}. Current tests may not adequately cover this change.`;
    } else if (entry.type === 'fixed' && coverageScore < 0.8) {
      gapType = 'missing-regression';
      recommendation = `Add specific regression test for bug fix: ${entry.description}`;
    } else {
      gapType = 'partial-coverage';
      recommendation = `Increase test coverage for: ${entry.description}. Currently at ${(coverageScore * 100).toFixed(0)}%.`;
    }

    const riskLevel = assessRisk(entry, coverageScore);

    gaps.push({ entry, gapType, riskLevel, recommendation });

    if (riskLevel === 'critical') {
      blockers.push(`${entry.type.toUpperCase()}: ${entry.description} - no adequate test coverage`);
    } else if (riskLevel === 'high') {
      warnings.push(`${entry.type.toUpperCase()}: ${entry.description} - insufficient test coverage`);
    }
  }

  const coveragePercentage =
    entries.length > 0 ? (coveredEntries / entries.length) * 100 : 100;

  const criticalUntested = gaps.some(
    (g) => g.riskLevel === 'critical' && thresholds.criticalRequireTests
  );

  const readyForRelease =
    coveragePercentage >= thresholds.minCoverage && !criticalUntested;

  const riskScore = calculateRiskScore(gaps, entries.length);

  return {
    version: entries[0]?.version || 'unknown',
    totalEntries: entries.length,
    coveredEntries,
    coveragePercentage: Math.round(coveragePercentage * 10) / 10,
    gaps,
    readyForRelease,
    riskScore,
    blockers,
    warnings,
  };
}

function assessRisk(
  entry: ChangelogEntry,
  coverageScore: number
): CoverageGap['riskLevel'] {
  if (entry.severity === 'critical' && coverageScore < 0.5) return 'critical';
  if (entry.breakingChange && coverageScore < 0.8) return 'critical';
  if (entry.type === 'security' && coverageScore < 0.8) return 'critical';
  if (entry.type === 'fixed' && coverageScore < 0.3) return 'high';
  if (entry.severity === 'major' && coverageScore < 0.5) return 'high';
  if (coverageScore < 0.3) return 'medium';
  return 'low';
}

function calculateRiskScore(gaps: CoverageGap[], totalEntries: number): number {
  if (totalEntries === 0) return 0;

  const riskWeights = { critical: 4, high: 2, medium: 1, low: 0.5 };
  const totalRisk = gaps.reduce((sum, gap) => sum + riskWeights[gap.riskLevel], 0);
  const maxRisk = totalEntries * riskWeights.critical;

  return Math.min(1, totalRisk / maxRisk);
}

Release Report Generator

// src/reporters/release-readiness-reporter.ts

import type { ReleaseReadiness } from '../analyzers/gap-detector';

export function generateReleaseReport(readiness: ReleaseReadiness): string {
  const lines: string[] = [];
  const status = readiness.readyForRelease ? 'READY' : 'NOT READY';

  lines.push(`# Release Readiness Report: v${readiness.version}`);
  lines.push('');
  lines.push(`**Status:** ${status}`);
  lines.push(`**Coverage:** ${readiness.coveragePercentage}% of changelog entries have mapped tests`);
  lines.push(`**Risk Score:** ${(readiness.riskScore * 100).toFixed(1)}%`);
  lines.push(`**Entries:** ${readiness.coveredEntries}/${readiness.totalEntries} covered`);
  lines.push('');

  if (readiness.blockers.length > 0) {
    lines.push('## Blockers');
    lines.push('');
    for (const blocker of readiness.blockers) {
      lines.push(`- ${blocker}`);
    }
    lines.push('');
  }

  if (readiness.warnings.length > 0) {
    lines.push('## Warnings');
    lines.push('');
    for (const warning of readiness.warnings) {
      lines.push(`- ${warning}`);
    }
    lines.push('');
  }

  if (readiness.gaps.length > 0) {
    lines.push('## Coverage Gaps');
    lines.push('');
    lines.push('| Change | Type | Risk | Gap Type | Recommendation |');
    lines.push('|--------|------|------|----------|----------------|');

    for (const gap of readiness.gaps) {
      const desc =
        gap.entry.description.length > 50
          ? gap.entry.description.substring(0, 50) + '...'
          : gap.entry.description;
      lines.push(
        `| ${desc} | ${gap.entry.type} | ${gap.riskLevel} | ${gap.gapType} | ${gap.recommendation.substring(0, 60)}... |`
      );
    }
  }

  return lines.join('\n');
}

CI Integration

GitHub Actions Release Gate

name: Release Readiness Check

on:
  pull_request:
    branches: [main]
    paths:
      - 'CHANGELOG.md'

jobs:
  check-readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Run changelog test mapping
        run: tsx scripts/map-release.ts
        env:
          MIN_COVERAGE: '80'
          REQUIRE_CRITICAL_TESTS: 'true'

      - name: Comment PR with readiness report
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            if (fs.existsSync('release-readiness-report.md')) {
              const report = fs.readFileSync('release-readiness-report.md', 'utf-8');
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.issue.number,
                body: report,
              });
            }

Release Mapping Script

// scripts/map-release.ts

import { readFileSync, writeFileSync } from 'fs';
import { execSync } from 'child_process';
import { parseKeepAChangelog } from '../src/parsers/changelog-parser';
import { mapChangesToTests } from '../src/mappers/file-change-mapper';
import { findSemanticMatches } from '../src/mappers/semantic-mapper';
import { analyzeReleaseReadiness } from '../src/analyzers/gap-detector';
import { generateReleaseReport } from '../src/reporters/release-readiness-reporter';
import { join } from 'path';

async function main(): Promise<void> {
  const changelogContent = readFileSync('CHANGELOG.md', 'utf-8');
  const changelog = parseKeepAChangelog(changelogContent);

  if (changelog.versions.length === 0) {
    console.log('No versions found in changelog');
    process.exit(0);
  }

  const latestVersion = changelog.versions[0];
  console.log(
    `Analyzing release v${latestVersion.version} (${latestVersion.entries.length} entries)`
  );