Goal

• Given information about failed tests <FAILURE_INFO> run the tests, understand what is the root cause, and create a plan to fix the tests

Workflow

Step 1: Run Tests

• The input <FAILURE_INFO> can be:

  • A failing test in pytest
    • E.g., dev_scripts_helpers/documentation/test/test_notes_to_pdf.py
  • A repro generated by pytest_failed_multi_build.py
    • E.g., tmp.pytest_failed_multi_build.repro.sh

• Gather information about the failing tests by:

  • Reading the code
  • Inspecting the content of <FAILURE_INFO>
  • Running the tests locally, if needed
    • E.g.,

      > pytest_log `<FAILURE_INFO>`
      > tmp.pytest_failed_multi_build.repro.sh
      

Step 2: Analyze Failures

• Read .claude/skills/pytest.rules.md about how to diagnose issues

• Collect the failing tests from the run

  > i pytest_failed
  ...
  Failed tests: 25/7
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_actions::test1
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_actions::test2
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_edge_cases::test1
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_edge_cases::test2
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_edge_cases::test3
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_edge_cases::test4
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_filters::test1
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_filters::test2
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_filters::test3
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_filters::test4
  dev_scripts_helpers/documentation/test/test_notes_to_pdf.py::Test_notes_to_pdf_filters::test5
  ...
  13:07:36 - WARN  lib_tasks_pytest.py pytest_failed:1745                 To run the failed tests run: tmp.pytest_failed.sh
  ...
  

• For the tests that involve running Docker / Apple containers run the same tests with different builds, as per ## Analyze the Effect of Container Builds in .claude/skills/pytest.rules.md

• Carefully consider if the root cause and the solution is the same for all the 3 builds

Step 4: Prepare A Plan to Fix Failures

• Prepare a plan following # Format for Unit Test Fixing Plan from .claude/skills/pytest.rules.md
• Save it in the file plan.triage_local_unit_tests.md in the current dir
• Do not print anything on screen

Step 5: Wait for User Feedback

• Do not make any change to the code, but only propose the fixes
• Ask the user to continue or not with the fixes

Step 6: Implement and Verify the Fix

• If the user says to continue, implement the fixes
• Verify that they work correctly as per # Verification of the Fix from .claude/skills/pytest.rules.md