Inside AI Code Explainers: AST Parsing, Tokenization, and Contextual Code Analysis

The Myth of the Raw Prompt

Most developers believe that an AI code explainer simply feeds raw source text into a large language model and asks: "What does this code do?" If you build a production developer tool that way, it fails almost immediately on non-trivial codebases. Passing unparsed text causes hallucinations: models confuse shadowed variable scopes, misunderstand asynchronous execution order, and fail to identify critical memory leaks.

Production-grade code explainers (such as those integrated into Cursor, GitHub Copilot, and Sourcegraph) work by fusing compiler theory with semantic retrieval. Before a single token is sent to an LLM, the source code is parsed into an Abstract Syntax Tree (AST), statically analyzed for control flow and complexity, and enriched with cross-file symbol definitions.

The 3-Phase Code Explanation Pipeline

To produce accurate, high-density explanations without hallucinations, the system runs through three concrete execution phases:

[Raw Source File] 
        │
        ▼
[Phase 1: Lexical Analysis & AST Generation] 
  - Tokenizes keywords, identifiers, and literals
  - Constructs recursive syntax tree (Babel / Tree-sitter / Roslyn)
        │
        ▼
[Phase 2: Static Analysis & Symbol Resolution]
  - Tracks variable scope bindings and closure captures
  - Calculates cyclomatic complexity and recursion depth
  - Resolves imported type definitions
        │
        ▼
[Phase 3: Contextual Augmentation & LLM Synthesis]
  - Builds structured AST summary prompt
  - Streams step-by-step logic breakdown with edge-case warnings

Understanding the Abstract Syntax Tree (AST)

An AST is a hierarchical tree representation of source code structure. It discards superficial formatting (whitespace, indentation, semicolons) and retains pure grammatical logic. Consider this simple JavaScript snippet:

function isAdult(age) {
  return age >= 18;
}

A compiler does not see five words. It builds an AST with specific typed nodes:

• FunctionDeclaration: Identifier isAdult, Param [Identifier: age]
• BlockStatement: Body containing one ReturnStatement
• BinaryExpression: Left [Identifier: age], Operator >=, Right [NumericLiteral: 18]

By operating on nodes instead of raw text, an explanation engine can immediately determine: "This function contains exactly one branch, accepts one numeric parameter, and returns a boolean value."

Production TypeScript: AST Code Analyzer

Here is a working Node.js script utilizing Babel parser and traverser. It inspects any JavaScript or TypeScript function, calculates its cyclomatic complexity, identifies external side effects, and prepares a deterministic explanation manifest:

// src/analyzer/ast-explainer.ts
import * as parser from '@babel/parser';
import traverse from '@babel/traverse';

export interface AnalysisResult {
  functionName: string;
  parameters: string[];
  cyclomaticComplexity: number;
  callsExternalFunctions: string[];
  hasAsyncAwait: boolean;
  hasTryCatch: boolean;
  summary: string;
}

export function analyzeFunctionAst(sourceCode: string): AnalysisResult {
  // 1. Parse code into an Abstract Syntax Tree
  const ast = parser.parse(sourceCode, {
    sourceType: 'module',
    plugins: ['typescript'],
  });

  let functionName = 'anonymous';
  const parameters: string[] = [];
  let complexity = 1; // Base complexity
  const externalCalls = new Set<string>();
  let hasAsync = false;
  let hasTryCatch = false;

  // 2. Traverse AST nodes with visitor pattern
  traverse(ast, {
    FunctionDeclaration(path) {
      functionName = path.node.id?.name ?? 'anonymous';
      hasAsync = path.node.async;
      path.node.params.forEach((param) => {
        if (param.type === 'Identifier') {
          parameters.push(param.name);
        }
      });
    },

    // Count decision points for cyclomatic complexity
    IfStatement() { complexity++; },
    ForStatement() { complexity++; },
    WhileStatement() { complexity++; },
    ConditionalExpression() { complexity++; }, // Ternary
    LogicalExpression(path) {
      if (path.node.operator === '&&' || path.node.operator === '||') {
        complexity++;
      }
    },

    TryStatement() {
      hasTryCatch = true;
    },

    CallExpression(path) {
      if (path.node.callee.type === 'Identifier') {
        externalCalls.add(path.node.callee.name);
      } else if (path.node.callee.type === 'MemberExpression' && path.node.callee.property.type === 'Identifier') {
        externalCalls.add(path.node.callee.property.name);
      }
    },
  });

  return {
    functionName,
    parameters,
    cyclomaticComplexity: complexity,
    callsExternalFunctions: Array.from(externalCalls),
    hasAsyncAwait: hasAsync,
    hasTryCatch,
    summary: `Function '${functionName}' accepts (${parameters.join(', ')}). Cyclomatic complexity: ${complexity}. External invocations: [${Array.from(externalCalls).join(', ')}].`,
  };
}

Synthesizing the Explanation with LLM Anchors

Once the AST analyzer extracts deterministic metadata, feed the structured payload into the language model alongside the source snippet. Below is the system prompt architecture that eliminates hand-waving explanations:

### System Prompt: Senior Systems Architect Explainer
You are an uncompromising senior software engineer reviewing code.
You receive two inputs:
1. Deterministic AST Analysis Metadata (Cyclomatic complexity, scope bindings, external calls)
2. Raw Source Code

Your mandate:
1. State what the function accomplishes in one unambiguous sentence.
2. Walk through control flow branch by branch, referencing specific AST nodes.
3. Flag any cyclomatic complexity greater than 5 as a refactoring candidate.
4. Call out uncaught exceptions, missing null checks, or unhandled promise rejections.
5. Do not use AI fluff words (no "delve", "leverage", "seamless"). Speak strictly in engineering facts.

AST vs. LLM Comparison Matrix

Detecting Hidden Edge Cases in Real Code

A great code explainer does not just restate what the code does line by line; it explains what happens when inputs deviate from the happy path. For instance, when analyzing this snippet:

function parseUserMetadata(rawJson: string) {
  const data = JSON.parse(rawJson);
  return data.user.profile.avatarUrl;
}

A basic prompt says: "This function parses JSON and retrieves the avatar URL." A compiler-backed explainer notes:

• Syntax Risk: JSON.parse throws an unhandled SyntaxError if rawJson contains malformed JSON or empty string inputs.
• Null Pointer Exception: Accessing nested properties without optional chaining (data?.user?.profile?.avatarUrl) throws TypeError: Cannot read properties of undefined (reading 'profile') if user is null.
• Type Safety: The return type implicitly resolves to any, disabling downstream compiler validation in TypeScript.

Summary: The Modern Explainer Stack

Building an elite code explanation tool requires respecting the compiler. By leveraging AST traversal, extracting scope graphs, and combining static metrics with constrained LLM generation, developers build tools that deliver deep, factual, and bug-preventing engineering insights.