Runbook: Generate Detection Report

Objective

To summarize the findings, logic, and performance of a specific detection rule or a set of related rules. This report is often used for periodic review, tuning documentation, reporting on detection capabilities, or providing context for incident investigations.

Scope

This runbook covers:

• Retrieving the definition and logic of specified detection rule(s).
• Gathering historical performance data (e.g., alert volume, severity, true/false positive rates if available from associated cases) for the rule(s) over a defined timeframe.
• Optionally gathering context from related SOAR cases or tuning documentation.
• Structuring the information into a standardized report format.
• Generating a Mermaid sequence diagram illustrating the data gathering process for the report.
• Writing the final Markdown report to a file.

This runbook explicitly excludes:

• Performing new validation or tuning steps for the detection rule (it reports on existing information).
• Implementing changes to the detection rule.

Inputs

• ${RULE_ID} or ${RULE_IDS}: The identifier(s) of the detection rule(s) to report on. This is mandatory.
• (Optional) ${REPORT_TIMEFRAME_DAYS}: Timeframe in days for gathering performance data (e.g., 30, 90). Defaults to 90 if not specified.
• (Optional) ${CASE_ID}: Relevant SOAR case ID if the report relates to a specific incident, tuning effort, or if case data is needed to determine TP/FP rates.
• (Derived) ${RULE_DEFINITION}: The logic/definition of the rule obtained from secops-mcp_list_security_rules.
• (Derived) ${PERFORMANCE_DATA}: Historical alert data and statistics for the rule from secops-mcp_get_security_alerts.
• (Derived) ${SOAR_CONTEXT}: Information from related SOAR cases obtained via soar-mcp_get_case_full_details.

Outputs

• ${REPORT_FILE_PATH}: The full path to the generated Markdown report file.
• ${REPORT_CONTENT}: The full Markdown content of the generated report.
• ${REPORT_GENERATION_STATUS}: Confirmation or status of the report file writing attempt.

Tools

• secops-mcp: list_security_rules, get_security_alerts (for performance data)
• secops-soar: get_case_full_details, post_case_comment (for context/documentation, though post_case_comment is not directly used in this workflow but might be used by a calling runbook)
• write_to_file (Replaces the conceptual write_report tool)

Workflow Steps & Diagram

1. Gather Rule Details: Retrieve the rule logic/definition for ${RULE_ID} (or each ID in ${RULE_IDS}) using secops-mcp_list_security_rules. Store in ${RULE_DEFINITION}.
2. Gather Performance Data: Retrieve historical alerts generated by the rule(s) over ${REPORT_TIMEFRAME_DAYS} (default 90) using secops-mcp_get_security_alerts. Analyze volume, severity, and potentially associated case statuses (TP/FP if ${CASE_ID} is provided and allows such correlation). Store in ${PERFORMANCE_DATA}.
3. Gather Context (Optional): If ${CASE_ID} is provided, review related SOAR case(s) using soar-mcp_get_case_full_details or tuning documentation if applicable. Store in ${SOAR_CONTEXT}.
4. Structure Report: Organize information according to a standard template (referencing rules-bank/reporting_templates.md). Key sections might include: Rule Details (ID, Name, Logic Summary from ${RULE_DEFINITION}), Performance Metrics (Alert Volume, TP/FP Ratio if known from ${PERFORMANCE_DATA} and ${SOAR_CONTEXT}), Key Findings/Observations, Tuning History/Recommendations (if available from ${SOAR_CONTEXT}).
5. Generate Mermaid Diagram: Create a Mermaid sequence diagram summarizing the tools used to gather data for this report.
6. Format Report: Compile the synthesized information and the Mermaid diagram into a final Markdown report. Store as ${REPORT_CONTENT}.
7. Write Report File: Save the report using write_to_file with path="./reports/detection_report_${RULE_ID}_${timestamp}.md" (adjust filename if multiple RULE_IDS) and content=${REPORT_CONTENT}. Store path in ${REPORT_FILE_PATH} and status in ${REPORT_GENERATION_STATUS}.

ADK Graph-Based Workflow Diagram

graph TD
    START(["START"]) --> extract_detection_report_payload_node["1. extract_detection_report_payload_node<br/><i>(Extract Rule & Scope Payload)</i>"]
    extract_detection_report_payload_node --> fetch_detection_stats_node["2. fetch_detection_stats_node<br/><i>(Fetch Historical Alert Stats & FP Rate)</i>"]
    fetch_detection_stats_node --> detection_report_router{"3. detection_report_router<br/><i>(Event.actions.route)</i>"}

    detection_report_router -- "HIGH_NOISE_LEVEL" --> handle_high_noise_branch["4a. handle_high_noise_branch<br/><i>(Document High Noise & Recommend Tuning)</i>"]
    detection_report_router -- "OPTIMAL_PERFORMANCE" --> handle_optimal_performance_branch["4b. handle_optimal_performance_branch<br/><i>(Document Optimal Rule Performance)</i>"]

    handle_high_noise_branch --> document_detection_report_node["5. document_detection_report_node<br/><i>(SOAR Comment & Report Summary)</i>"]
    handle_optimal_performance_branch --> document_detection_report_node

Sequence Diagram

sequenceDiagram
    participant Analyst/User
    participant AutomatedAgent as Automated Agent (MCP Client)
    participant SIEM as secops-mcp
    participant SOAR as secops-soar

    Analyst/User->>AutomatedAgent: Generate Detection Report\nInput: RULE_ID, REPORT_TIMEFRAME_DAYS, CASE_ID (opt)

    %% Step 1: Gather Rule Details
    AutomatedAgent->>SIEM: list_security_rules(rule_id=RULE_ID) %% or iterate if RULE_IDS
    SIEM-->>AutomatedAgent: Rule Definition/Logic (RULE_DEFINITION)

    %% Step 2: Gather Performance Data
    AutomatedAgent->>SIEM: get_security_alerts(rule_id=RULE_ID, hours_back=REPORT_TIMEFRAME_DAYS*24)
    SIEM-->>AutomatedAgent: Historical Alert Data (PERFORMANCE_DATA)
    Note over AutomatedAgent: Analyze performance metrics

    %% Step 3: Gather Context (Optional)
    opt Case ID Provided
        AutomatedAgent->>SOAR: get_case_full_details(case_id=CASE_ID)
        SOAR-->>AutomatedAgent: Case Context / Tuning Notes (SOAR_CONTEXT)
    end

    %% Step 4 & 5: Structure Report & Generate Diagram
    Note over AutomatedAgent: Organize report sections (Rule Details, Performance, Findings...)
    Note over AutomatedAgent: Create Mermaid diagram summarizing report generation steps

    %% Step 6 & 7: Format & Write Report
    Note over AutomatedAgent: Compile final Markdown content (REPORT_CONTENT)
    AutomatedAgent->>AutomatedAgent: write_to_file(path="./reports/detection_report_${RULE_ID}_${timestamp}.md", content=REPORT_CONTENT)
    Note over AutomatedAgent: Report file created (REPORT_FILE_PATH, REPORT_GENERATION_STATUS)

    AutomatedAgent->>Analyst/User: attempt_completion(result="Detection report generated for RULE_ID. Path: REPORT_FILE_PATH")

Completion Criteria

• The definition and logic for the specified detection rule(s) (${RULE_ID} or ${RULE_IDS}) have been retrieved.
• Historical performance data for the rule(s) over the specified timeframe has been gathered and analyzed.
• Optional context from SOAR cases or tuning documentation has been incorporated if provided.
• A comprehensive Markdown report summarizing the rule details, performance, and any relevant context has been structured and formatted, including a Mermaid diagram of the data gathering process.
• The final report has been successfully written to a file, and its path (${REPORT_FILE_PATH}) and generation status (${REPORT_GENERATION_STATUS}) are available.

Rubrics

The following rubric is used to evaluate the execution of this Reporting runbook by an LLM agent.

Grading Scale (0-100 Points)

Criteria	Points	Description
Data Collection	25	Gathered all necessary data points and metrics for the report.
Report Generation	30	Generated the report in the correct format with accurate content.
Quality & Clarity	15	Ensure the report is readable, well-structured, and error-free.
Delivery	15	Delivered or saved the report to the correct location/recipient.
Operational Artifacts	15	Produced required artifacts: Sequence diagram, execution metadata (date/cost), and summary.

Evaluation Criteria Details

1. Data Collection (25 Points)

• 25 pts: Successfully retrieved all required data (alerts, stats, summaries) from sources.

2. Report Generation (30 Points)

• 15 pts: Formatted the data correctly into the target template (Markdown, PDF, etc.).
• 15 pts: Included all required sections (Executive Summary, Details, etc.).

3. Quality & Clarity (15 Points)

• 15 pts: The generated text is coherent, accurate, and professional.

4. Delivery (15 Points)

• 15 pts: Successfully saved the file or sent the notification/email as required.

5. Operational Artifacts (15 Points)

• 5 pts: Sequence Diagram: Produced a Mermaid sequence diagram visualizing the steps taken.
• 5 pts: Execution Metadata: Recorded the date, duration, and estimated token cost.
• 5 pts: Summary Report: Generated a concise summary of the actions and outcomes.