GitHub PR Reviewer Automation

Create a cron automation that watches one or more GitHub repositories for pull requests with a review trigger label, starts an OpenHands review conversation once per label event, and publishes the AI review to GitHub. Windows PowerShell equivalents for the setup, packaging, upload, and API-check shell snippets are in references/windows.md.

The automation script is deterministic: PR discovery, label-event tracking, state persistence, stale-result suppression, the repository checkout, and its removal are all handled in Python. The LLM is invoked only for the review itself.

The script prepares each review's workspace before the agent starts: the pull request's head commit is downloaded as a tarball and extracted to a directory of its own, which becomes the conversation's working directory. The agent is told not to clone, fetch, check out, or delete anything, and the script removes the checkout once the conversation has stopped. Nothing accumulates between runs.

---

Prerequisites

Required secret

Verify that the following secret is set in OpenHands Settings -> Secrets:

Secret name	Token type	Minimum permissions
GITHUB_PERSONAL_ACCESS_TOKEN	Classic PAT	repo for private repos or public_repo for public repos
GITHUB_PERSONAL_ACCESS_TOKEN	Fine-grained PAT	Contents: Read, Metadata: Read, Pull requests: Read and Write, Issues: Read and Write

Pull-request write access is required because the agent publishes a pull request review, not just an issue comment. A token with only Pull requests: Read will poll happily and then fail at the point of publishing.

When several repositories are monitored, the token must cover all of them.

Check with:

curl -s https://api.github.com/user \
  -H "Authorization: Bearer $GITHUB_PERSONAL_ACCESS_TOKEN" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('login') or d.get('message'))"

If the token is missing or invalid, inform the user and stop.

---

Setup Workflow

Follow these steps in order.

Step 1 - Verify GITHUB_PERSONAL_ACCESS_TOKEN

Run the curl check above.

• If absent: "GITHUB_PERSONAL_ACCESS_TOKEN is not set. Please add it in OpenHands Settings -> Secrets." Stop.
• If the API returns {"message": "Bad credentials"}: tell the user the token is invalid and ask them to update it. Stop.

Step 2 - Collect repositories

Ask: "Which GitHub repositories should be monitored? (Format: owner/repo, e.g. myorg/backend. List several separated by commas to review them all from one automation.)"

Validate access to each repository:

curl -s "https://api.github.com/repos/{owner}/{repo}" \
  -H "Authorization: Bearer $GITHUB_PERSONAL_ACCESS_TOKEN" \
  | python3 -c "
import json, sys
d = json.load(sys.stdin)
if 'message' in d:
    print('ERROR:', d['message'])
else:
    print(f\"Accessible. Private: {d.get('private')}. Permissions: {d.get('permissions')}\")
"

Record every accepted repository into REPOS = ["{owner}/{repo}", ...]. If one repository fails the check, say which and ask whether to continue without it.

Each repository is polled independently and keeps its own state, so pull-request numbers never collide between them. The trigger label, tone, and schedule are shared by all of them; a repository needing different settings wants its own automation.

Step 3 - Collect trigger label

Ask: "Which PR label should trigger a review? (Press Enter for the default: openhands-review.)"

Record the answer as TRIGGER_LABEL. If the label does not exist yet, tell the user that GitHub will still record the event once the label is created and applied to a PR.

The automation reviews a PR when it sees the latest matching labeled event for that label. To request another review later, remove and re-apply the label.

Step 4 - Collect review tone

Ask: *"What review tone should the reviewer use?

1. Thorough (default) - comprehensive coverage of correctness, security, tests, style
2. Concise - high-signal only, skips minor style feedback
3. Friendly - constructive and encouraging (Press Enter for Thorough, or type your choice or any custom style description)"*

Map the choice to REVIEW_TONE:

Answer	REVIEW_TONE	REVIEW_STYLE_INSTRUCTIONS
1 / Enter	"thorough"	""
2	"concise"	""
3	"friendly"	""
Custom text, e.g. strict but kind	"thorough"	the custom text verbatim

Step 5 - Collect cron schedule

Ask: "How often should the automation poll for labeled PRs? (Press Enter for the default: every 5 minutes. Use a cron expression for a different interval, e.g. 0 * * * * = hourly)"

Default: */5 * * * *.

Record as CRON_SCHEDULE.

Step 6 - Generate the automation script

Read scripts/main.py from this skill's directory. Apply exactly six constant substitutions near the top of the file:

The script also reads a config.json shipped beside it, if there is one, over these constants. That is how the catalog entry (automations/catalog/github-pr-reviewer/) configures an unmodified copy, since a declarative host cannot rewrite Python. This setup path substitutes the constants and ships no config.json, so the two never collide.

Placeholder	Replace with
REPOS = ["owner/repo"]	REPOS = ["{owner_repo}", ...] - one entry per repository collected in Step 2
TRIGGER_LABEL = "openhands-review"	TRIGGER_LABEL = "{trigger_label}"
REVIEW_TONE = "thorough"	REVIEW_TONE = "{review_tone}"
REVIEW_STYLE_INSTRUCTIONS = ""	REVIEW_STYLE_INSTRUCTIONS = "{style_instructions}"
REPO_REVIEW_GUIDE_PATH = ".agents/skills/custom-codereview-guide.md"	leave unchanged to auto-load a repo review guide at this path, or set to "" to disable
DEFAULT_OPENHANDS_URL = "http://localhost:8000"	leave unchanged unless the user has a preference

Use a safe string writer such as json.dumps(value) when inserting user-provided repository names, labels, or style instructions into Python string literals. json.dumps(list_of_repos) produces the whole REPOS list safely in one step.

Write the customized script to a temporary build directory:

mkdir -p /tmp/pr-reviewer-build
# write the customized main.py to /tmp/pr-reviewer-build/main.py

Validate syntax before packaging:

python3 -m py_compile /tmp/pr-reviewer-build/main.py && echo "Syntax OK"

Fix any syntax errors before proceeding.

Step 7 - Package and upload

Determine the Automation backend URL and auth from the <RUNTIME_SERVICES> block in your system context:

• OPENHANDS_HOST: the Automation backend url_from_agent
• Auth: X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY

tar -czf /tmp/pr-reviewer.tar.gz -C /tmp/pr-reviewer-build .

TARBALL_PATH=$(curl -s -X POST \
  "${OPENHANDS_HOST}/api/automation/v1/uploads?name=github-pr-reviewer" \
  -H "X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY" \
  -H "Content-Type: application/gzip" \
  --data-binary @/tmp/pr-reviewer.tar.gz \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['tarball_path'])")

echo "Uploaded: $TARBALL_PATH"

Step 8 - Register the automation

curl -s -X POST "${OPENHANDS_HOST}/api/automation/v1" \
  -H "X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"GitHub PR Reviewer: {repo_summary} label {trigger_label}\",
    \"trigger\": {\"type\": \"cron\", \"schedule\": \"{cron_schedule}\"},
    \"tarball_path\": \"$TARBALL_PATH\",
    \"entrypoint\": \"python3 main.py\",
    \"timeout\": 600
  }" | python3 -m json.tool

Use the single repository as {repo_summary} when there is one, and something like 3 repos when there are several. A poll now downloads a tarball per queued review, so the timeout allows for that; a run never waits for a review to finish, only for it to be started.

Record the returned id.

Step 9 - Confirm

Tell the user:

✅ GitHub PR Reviewer is running!

• Automation ID: {id}
• Repositories: {owner}/{repo}, ... (one line each)
• Trigger label: {trigger_label}
• Review tone: {tone}
• Polling schedule: {cron_schedule}
• State file per repository: ~/.openhands/workspaces/automation-state/github_pr_reviewer_label_event_{id}_{owner}__{repo}.json

Apply the {trigger_label} label to a pull request to queue a review. Each label event is processed once. To request another review, remove and re-apply the label.

The review is published as a pull request review on the head commit, with inline comments where a finding maps to a changed line.

---

Runtime Behaviour (per poll)

Each cron run executes main.py, which resolves and validates GITHUB_PERSONAL_ACCESS_TOKEN once, then processes every repository in REPOS independently. One repository failing does not stop the others; the run fails only if every repository fails.

For each repository:

1. Loads that repository's state (see references/state-schema.md).
2. Verifies repository access.
3. Lists open PRs, newest-updated first.
4. For each open PR carrying TRIGGER_LABEL:
   • Refetches current PR metadata to avoid acting on stale list data.
   • Finds the latest matching GitHub labeled issue event.
   • Skips the event if it has already been tracked.
   • Downloads the PR's head commit as a tarball and extracts it to {WORKSPACE_BASE}/repositories/{owner}__{repo}/pr-{number}-{sha12}. The archive is checked as it is unpacked: a single root, no absolute or .. paths, and symlinks skipped rather than materialised.
   • Starts an OpenHands conversation whose working directory is that checkout, with a review prompt carrying PR metadata, the exact head SHA, and label event details.
   • Posts an acknowledgement comment with the label event, head SHA, and conversation link.
   • Records the review in state with status: "active" and the checkout path.
   • If the checkout or the conversation cannot be created, the checkout is removed and nothing is recorded, so the next poll retries the label event.
5. For each active review conversation:
   • Marks it closed without posting if the PR has closed or merged.
   • Suppresses stale results if the PR head SHA changed after the review was queued.
   • When the conversation reaches idle, finished, error, or stuck, asks GitHub whether a review by the token's own user exists for that head SHA. If it does, the review is complete. If it does not, the agent's final response is posted as a comment so the work is not lost.
   • Abandons a conversation that has not reached a terminal status within two hours, so its checkout can be reclaimed.
6. Removes the checkout of every finished review, but only after confirming the conversation has stopped - deleting it under a running agent would remove its working directory. When that cannot be confirmed the directory is left alone and the next poll tries again.
7. Saves that repository's state atomically.

The completion callback fires once for the whole run.

---

Additional Resources

• references/state-schema.md - State JSON schema, field definitions, and review lifecycle diagram.
• scripts/main.py - The complete automation script. Customize the five constants at the top before packaging.
• tests/test_main.py - Unit tests for the checkout, its removal, and state handling. Run them from the skill root with python -m pytest tests/ after editing the script.

---

Troubleshooting

Symptom	Likely cause	Fix
Bot never queues reviews	Trigger label not present or no matching labeled event	Apply the configured label to the PR
"Bad credentials" in run logs	Token expired	Rotate and update GITHUB_PERSONAL_ACCESS_TOKEN
404 on repo access	Repo name wrong or no access	Re-check the entry in REPOS and the token's permissions
One repository is skipped, others work	That repository failed its access check	Read the === owner/repo === block in the run log
Same PR not reviewed after new commits	Label event was already processed	Remove and re-apply the trigger label
Review result never posts	Conversation still running or stuck	Open the conversation link from the acknowledgement comment
Stale review suppressed	PR head SHA changed while the agent was reviewing	Re-apply the trigger label after the latest commit
Review arrives as a plain comment, not a review	Publishing failed, so the script posted the text as a fallback	Check that the token has Pull requests: Read and Write
Agent reports it cannot clone the repo	Prompt asked it not to; the workspace is already the checkout	No action - the code is at the head SHA in its working directory
Checkouts remain under repositories/	Their conversations had not stopped yet	They are removed by a later poll once the conversation is terminal