Automated PR Reviews with GitHub Actions: Building an AI Code Reviewer

The Frustration with Noisy Review Bots

Most engineering teams suffer from pull request review bottlenecks. Pull requests sit open for days waiting for senior developer eyes. To solve this, teams often install generic AI review marketplace bots, only to regret it within forty-eight hours. These bots spam pull requests with thirty comments pointing out trailing whitespace, rephrasing comments that already explain themselves, and praising trivial variable renames.

Developers quickly develop alert blindness and ignore the bot entirely. An automated code reviewer is only valuable if it respects developer attention: it must stay silent when code is clean, ignore generated artifacts like lockfiles, and comment strictly on concrete architectural hazards such as race conditions, unindexed database queries, credential leaks, and broken error propagation. Here is how to construct your own custom, high-signal PR review GitHub Action using TypeScript and Octokit.

Architecture: Filtering, Diff Parsing, and Inline Placement

A production-ready review action does not post generic top-level issue comments. It inspects unified diff patches, maps security or architectural findings to exact line numbers on the modified file, and submits a structured inline review.

[GitHub PR Event: opened or synchronize]
                  │
                  ▼
[.github/workflows/ai-review.yml]
                  │
                  ▼
[Diff Extractor (Octokit)] ───► Filters out lockfiles, svgs, build assets
                  │
                  ▼
[LLM Structured Reviewer]  ───► Evaluates git patch hunks with strict schema
                  │
                  ▼
[Octokit Review Submitter] ───► POST /repos/{owner}/{repo}/pulls/{id}/reviews
                                (Attaches inline line-specific comments)

Step 1: The Least-Privilege GitHub Action Workflow

Never give CI review bots administrative access. A pull request reviewer only needs read permissions on repository contents and write permissions on pull requests.

# .github/workflows/ai-code-review.yml
name: AI Automated Code Review

on:
  pull_request:
    types: [opened, synchronize]
    paths-ignore:
      - '**.lock'
      - 'package-lock.json'
      - 'pnpm-lock.yaml'
      - '**.min.js'
      - '**.svg'
      - 'docs/**'

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install Reviewer Dependencies
        run: |
          cd .github/scripts/reviewer
          npm ci

      - name: Execute Pull Request Review
        run: node .github/scripts/reviewer/dist/index.js
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Step 2: Parsing Git Diffs and Filtering Noise

We build our action in TypeScript using @octokit/rest. First, we retrieve the pull request files and filter out files that language models should never inspect.

// .github/scripts/reviewer/src/index.ts
import { Octokit } from '@octokit/rest';
import { createAnthropic } from '@ai-sdk/anthropic';
import { generateObject } from 'ai';
import { z } from 'zod';
import * as github from '@actions/github';

const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const IGNORED_EXTENSIONS = ['.json', '.lock', '.md', '.png', '.jpg', '.svg', '.map'];

interface InlineComment {
  path: string;
  line: number;
  body: string;
}

async function run(): Promise<void> {
  const context = github.context;
  if (!context.payload.pull_request) {
    console.log('No pull request found in context. Skipping.');
    return;
  }

  const pullNumber = context.payload.pull_request.number;
  const { owner, repo } = context.repo;

  console.log(`Analyzing PR #${pullNumber} on ${owner}/${repo}`);

  // 1. Fetch changed files
  const { data: files } = await octokit.rest.pulls.listFiles({
    owner,
    repo,
    pull_number: pullNumber,
    per_page: 100,
  });

  const inlineComments: InlineComment[] = [];

  for (const file of files) {
    if (file.status === 'removed' || !file.patch) continue;
    if (IGNORED_EXTENSIONS.some((ext) => file.filename.endsWith(ext))) continue;
    if (file.changes > 300) {
      console.log(`Skipping large file diff: ${file.filename} (${file.changes} changes)`);
      continue;
    }

    const issues = await analyzeFileDiff(file.filename, file.patch);
    for (const issue of issues) {
      inlineComments.push({
        path: file.filename,
        line: issue.line,
        body: `**[AI Review - ${issue.severity}]** ${issue.comment}`,
      });
    }
  }

  if (inlineComments.length === 0) {
    console.log('No critical defects detected. Pull request approved silently.');
    return;
  }

  // 2. Submit batched inline review
  await octokit.rest.pulls.createReview({
    owner,
    repo,
    pull_number: pullNumber,
    event: 'COMMENT',
    body: '### Automated Architectural Review\nI identified potential defects in the modified files. Please review the line annotations below.',
    comments: inlineComments.map((c) => ({
      path: c.path,
      line: c.line,
      body: c.body,
    })),
  });

  console.log(`Submitted review with ${inlineComments.length} inline annotations.`);
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});

Step 3: Enforcing Schema-Driven Review Evaluations

To prevent conversational filler like "Great job on this line!", we force the model to output a structured array conforming to a Zod schema. If the code is solid, the array returns empty.

// .github/scripts/reviewer/src/analyzer.ts
const ReviewSchema = z.object({
  defects: z.array(
    z.object({
      line: z.number().describe('The precise line number in the NEW modified file where defect occurs'),
      severity: z.enum(['CRITICAL', 'WARNING', 'SECURITY']),
      comment: z.string().describe('Concise technical explanation of the defect and recommended fix'),
    })
  ),
});

export async function analyzeFileDiff(filename: string, patch: string) {
  const systemPrompt = `You are a Principal Software Security and Performance Reviewer.
Your job is to inspect git diff patches and report ONLY genuine bugs, memory leaks, concurrency races, or security flaws.

Strict Rules:
1. Never comment on formatting, whitespace, or variable naming preferences.
2. Never comment with praise or greetings.
3. The 'line' number must match an added line ('+') present in the patch.
4. If no severe bugs exist, return an empty 'defects' array.`;

  const prompt = `Inspect this unified patch for file: ${filename}

\`\`\`diff
${patch}
\`\`\``;

  const result = await generateObject({
    model: anthropic('claude-3-5-sonnet-20241022'),
    schema: ReviewSchema,
    system: systemPrompt,
    prompt,
  });

  return result.object.defects;
}

Testing with a Real Concurrency Defect

Consider a developer submitting a pull request that modifies an account balance handler in TypeScript:

// src/services/account.ts
+ export async function withdrawFunds(userId: string, amount: number) {
+   const user = await db.user.findUnique({ where: { id: userId } });
+   if (user.balance >= amount) {
+     await db.user.update({
+       where: { id: userId },
+       data: { balance: user.balance - amount }
+     });
+   }
+ }

When the GitHub Action runs, it ignores the trivial imports and fires an inline comment directly on line 6:

**[AI Review - CRITICAL]** Potential race condition (Time-of-Check to Time-of-Use). Two concurrent withdrawal requests can pass the balance check before either update completes. Use an atomic database decrement or an isolated transaction block: `data: { balance: { decrement: amount } }`.

Three Operational Realities to Master

Running an action on every pull request commit requires handling production friction:

• GitHub API Rate Limiting: A large monorepo with 50 pull requests an hour can burn through secondary API rate limits. Cache commit SHAs so you never re-analyze a commit that was already reviewed.
• Diff Line Offset Calculation: GitHub's review API requires that the line parameter matches an added line in the new version of the file. If you point to a deleted line, Octokit throws an HTTP 422 Unprocessable Entity (line must be part of the diff) error. Always validate that candidate line numbers exist in the patch's addition hunks.
• Context Window Slicing: Huge diffs with hundreds of modified lines will exceed prompt token limits. Chunk multi-hundred line diffs into per-hunk queries to maintain evaluation depth.

A custom GitHub Action gives your team the benefits of continuous automated review without third-party vendor lock-in or noisy notifications.

For more on integrating AI into your local terminal workflows, check our breakdown on modern AI coding workflows: CLI tools and prompt architecture, explore compiler AST inspections in inside AI code explainers, and compare toolchains in best AI code generators.