create-api-test-script

Arguments

Parse $ARGUMENTS as the target to test, such as web/search, images/search, brave, or context7.

If the target is missing or ambiguous, ask which app skill or provider the script should exercise.

Instructions

You are creating a manual backend API/app-skill test script that may need Vault-backed provider secrets. Use this workflow for quick probes, provider evaluations, and reproducible integration checks that are not part of the normal pytest suite.

Step 1: Choose The Script Location

Put Vault-dependent scripts under backend/scripts/ so they are available inside the running api container as /app/backend/scripts/<script>.py.

Do not put Vault-dependent scripts under scripts/api_tests/ unless the script is intentionally host-only. The repo-level scripts/ directory is not mounted into the api container by default, so those scripts cannot directly use /vault-data/api.token or the container runtime environment.

Step 2: Choose The Execution Path

Use the path that matches what you are testing:

Test target	Recommended path	Why
App skill behavior	SkillRegistry.dispatch_skill()	Matches the in-process app-skill architecture used by API and workers
Provider wrapper behavior	Direct provider import + SecretsManager	Isolates provider auth, request, parsing, and errors
Public REST API contract	Authenticated /v1/apps/{app}/skills/{skill}	Tests external API auth, billing, and response envelope

Avoid old per-app container endpoints such as http://app-web:8000/skills/search. Apps now load in-process inside api and Celery workers.

Step 3: Use The Container Command

Run Vault-dependent scripts with:

docker exec api python /app/backend/scripts/<script>.py

Use python, not host python3, when documenting the in-container command. The script itself may use a #!/usr/bin/env python3 shebang, but the command should be explicit and container-local.

Step 4: App Skill Script Template

Use this when testing the actual app skill dispatch path:

#!/usr/bin/env python3
"""
Purpose: Test <app>/<skill> through the in-process app skill registry.
Architecture: Runs inside the api container where Vault and backend apps are available.
Data sources: Existing OpenMates app skill and its configured providers.
Tests: Manual CLI verification with small, low-cost sample requests.
Usage: docker exec api python /app/backend/scripts/test_<app>_<skill>.py
"""

from __future__ import annotations

import asyncio
import json
import logging
import sys
from typing import Any

from backend.core.api.app.services.skill_registry import build_skill_registry


APP_ID = "<app>"
SKILL_ID = "<skill>"


def summarize_response(response: dict[str, Any]) -> dict[str, Any]:
    results = response.get("results") or []
    first_group = results[0] if results else {}
    first_items = first_group.get("results") or [] if isinstance(first_group, dict) else []
    return {
        "provider": response.get("provider"),
        "error": response.get("error"),
        "group_count": len(results),
        "result_count": len(first_items),
        "first_title_present": bool(first_items and first_items[0].get("title")),
    }


async def main() -> int:
    logging.basicConfig(level=logging.WARNING)
    registry, metadata = build_skill_registry()
    if APP_ID not in metadata:
        print(json.dumps({"status": "fail", "error": f"app not loaded: {APP_ID}"}))
        return 1
    if not registry.is_skill_available(APP_ID, SKILL_ID):
        print(json.dumps({"status": "fail", "error": f"skill not available: {APP_ID}/{SKILL_ID}"}))
        return 1

    response = await registry.dispatch_skill(
        APP_ID,
        SKILL_ID,
        {"requests": [{"id": "probe", "query": "OpenMates open source AI assistant", "count": 1}]},
    )
    summary = summarize_response(response)
    status = "pass" if summary["result_count"] > 0 and not summary["error"] else "fail"
    print(json.dumps({"status": status, **summary}, ensure_ascii=False))
    return 0 if status == "pass" else 1


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Adjust request fields to the target skill's tool_schema. For non-search skills, keep the summary similarly small and non-sensitive.

Step 5: Provider Script Template

Use this when testing a provider wrapper directly:

#!/usr/bin/env python3
"""
Purpose: Test the <provider> provider wrapper with Vault-backed credentials.
Architecture: Runs inside the api container and uses SecretsManager.
Data sources: External <provider> API through backend/shared/providers.
Tests: Manual CLI verification with one low-cost sample request.
Usage: docker exec api python /app/backend/scripts/test_<provider>_provider.py
"""

from __future__ import annotations

import asyncio
import json
import sys

from backend.core.api.app.utils.secrets_manager import SecretsManager


async def main() -> int:
    secrets_manager = SecretsManager()
    await secrets_manager.initialize()
    try:
        # Import and call the provider here. Keep counts small.
        # Never print secrets, prefixes, lengths, tokens, or raw auth headers.
        result = {"replace": "with provider result"}
    finally:
        await secrets_manager.aclose()

    print(json.dumps({"status": "pass", "result": result}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Step 6: Output Rules

Print only non-sensitive verification data:

• status: pass or fail
• provider name
• result counts
• selected non-sensitive fields, such as whether a title or URL is present
• error messages that do not include credentials

Do not print:

• API keys, tokens, OAuth credentials, cookies, or Vault values
• key prefixes, suffixes, lengths, masked values, or auth headers
• private user data or raw logs with identifiers
• large raw provider payloads unless the user explicitly needs a fixture and the payload is reviewed for sensitive content

If saving fixtures, write them under backend/scripts/results/ or another explicit non-source output directory, and keep them out of commits unless the user asks to add sanitized fixtures.

Step 7: Verify

Run the script in the api container:

docker exec api python /app/backend/scripts/<script>.py

If it fails because the provider is rate-limited or unavailable, report that separately from script correctness. Do not increase request counts to brute-force provider failures.

Step 8: Cleanup And Commit Scope

Before committing, ensure the commit includes only the script and any intentional docs/skill changes. Do not commit generated result JSON unless it is a sanitized fixture requested by the user.

Rules

• New .py scripts need a 5-10 line file header comment or docstring.
• Default to the smallest possible external request count, usually count=1.
• Use asyncio and httpx-based provider code; do not add requests.
• Prefer SkillRegistry.dispatch_skill() for app-skill tests.
• Prefer direct provider calls only for provider-specific tests.
• Never use old app-{id}:8000/skills/... endpoints for new scripts.
• Never expose secrets or partial secrets in logs, output, fixtures, or comments.