GitHubScanner

Overview

GitHubScanner (The Hartford) is an AI security reviewer for repositories, built on Google ADK and Gemini.

• How a scan runs. You point it at a repo. A root agent clones it, detects the stack, and delegates to specialist agents:
  • a planner
  • a security agent: OWASP / CWE, with Java and Spring-specific patterns
  • a dependency agent: known CVEs from OSV, exploit likelihood from FIRST's EPSS, plus end-of-life, outdated packages and typosquats
  • a report agent
  • a remediation agent, on demand
• Trust is the hard part. LLMs invent findings, so every finding passes deterministic validators before anyone sees it.
• Fixes. They're generated in a throwaway clone and exported as patches for a developer to review. Nothing is pushed.
• Tests. About 3,400, at 86% coverage.

System

CVE enrichment

• OSV is the primary source because it knows packages and version ranges. It maps straight to pom.xml, package.json and lockfiles.
• CVSS vs EPSS. CVSS says how bad a vulnerability would be. EPSS says how likely it is to be exploited in the next 30 days.
  • A CVSS 9.8 with EPSS 0.001 can wait.
  • A CVSS 7.5 with EPSS 0.9 is today's problem.

Trust pipeline: defending against hallucinated findings

• Why deterministic checks instead of an LLM judge? A judge model shares the generator's failure modes and doubles the cost. String and hash checks are cheap, reproducible and auditable.
• The 5% figure is a gate, not an accuracy claim. A scan is flagged when more than 5% of its findings fail grounding checks.
• Next refinement: re-read the file and assert that the cited snippet appears within ±N lines of the claimed line. That single check removes most fabricated findings.

Threat model: OWASP Top 10 for LLM applications, applied to the scanner itself

The scanned repository is attacker-controlled input, so the scanner has to be threat-modeled like any other LLM application.

Secrets and identity.

• It uses a GitHub App, not a personal access token, for short-lived, repo-scoped access.
• The App's private key lives in Secret Manager.
• The service mints an RS256 JWT (cached 9 min) and exchanges it for an installation token (cached 1 h).
• No long-lived credential is written to a repo or a log line.

Cloud design inside the app

• Environment detection. K_SERVICE set means it's running on Cloud Run: the config JSON loads from Secret Manager. Otherwise it reads .env.
• Reports. Written to a GCS bucket and served through 24-hour V4 signed URLs. No public bucket and no auth proxy.
• Scan state: SQLite synced to GCS. Syncs run at startup, every 5 min, after each scan, and at shutdown.
  • This forces max-instances=1. GCS FUSE and Litestream were considered and rejected.
  • To scale out, move to Cloud SQL Postgres / AlloyDB or Firestore.
• Long scans. 330–411 s measured on WebGoat and Spring Boot 2.6, which is past Cloud Run's 300 s default request timeout.
  • The production pattern is async: POST /scan returns a scan_id, Cloud Tasks / Pub/Sub hands it to a Cloud Run Job, status goes in the DB, and the client polls or gets a webhook.
• Region pinning. GOOGLE_CLOUD_LOCATION is pinned during a scan, because Gemini model availability differs by region.

Deployment: AWS CI/CD → GCP Cloud Run, keyless

The enterprise pipeline runs in AWS and the runtime is GCP. The two are bridged with Workload Identity Federation, so no GCP service-account keys exist anywhere in AWS.

• Keyless cross-cloud auth. CodeBuild's AWS role credentials are exchanged at GCP STS for a short-lived token that impersonates a single deploy service account. There are no JSON keys to leak or rotate.
• Immutable, traceable images. Tagged latest, a semantic version and the git SHA, so any SHA can be rolled back to.
• Shift-left gates. SAST, OSS licence and vulnerability checks, and an image scan all run before any image reaches a registry.

The same deployment on AWS and OCI

The model layer sits behind a factory, so it's model-agnostic: Vertex vs AI Studio is a config switch. Pointing it at a privately hosted endpoint inside the VPC is also a config change. For regulated or defense customers, customer code never goes to a public model API.

Scaling out and testing

• Org-wide scanning:
  • A GitHub webhook (push or PR) feeds a queue, and a pool of scan workers (Cloud Run Jobs or Kubernetes Jobs) writes results to Postgres.
  • The CVE cache becomes shared, which removes the single-instance constraint.
  • Scans are deduplicated by commit SHA, and only changed files plus the dependency diff are re-scanned.
• SAST + LLM, not either/or.
  • Regex "hint" tools run first, and the LLM judges them.
  • SAST is precise on known patterns. The LLM adds context: is this permitAll on a sensitive route? Do two medium issues chain into a critical one?
• Testing an LLM agent:
  • conftest.py mocks google.adk globally.
  • Tests cover tools, parsers, validators, grading math, event extraction and prompt content: the prompt must contain the grounding rules.
  • Canned JSON fixtures stand in for model output, including malformed output.
  • Plus integration and end-to-end runs against WebGoat and Spring Boot 2.6.

Engineering lessons:

1. LLM "fixes" overwrote whole files with a snippet.
   • The model returned only the changed lines, and the writer replaced the file.
   • Fix: size guards, and read-only by default for local paths.
   • Validate LLM output shape like user input.
2. Re-scans were non-deterministic, so "is it fixed?" had no answer.
   • Replaced with verification mode: deterministic IDs and 3-level fingerprint matching.
   • If the scanner can't prove an issue is fixed, it stays open.
3. Gemini 3 looped at temperature < 1.0.
   • Temperature was pinned to 1.0.
   • Thinking budgets moved into the planner.
4. Enterprise network controls blocked Vertex AI, and proxies broke HTTP/2.
   • An error classifier now reports these as infrastructure errors instead of "no findings".

Acronyms

Underlined acronyms show these definitions on hover or tap. Full glossary →

ADK

Agent Development Kit Google's open-source framework for building agents and multi-agent systems. ReliBot and GitHubScanner run on it.

ALB

Application Load Balancer AWS's HTTP load balancer. It can enforce OIDC login itself.

API

Application Programming Interface How programs call each other, such as a REST endpoint.

AWS

Amazon Web Services Amazon's cloud. GitHubScanner's CI/CD pipeline runs in AWS.

CI

Continuous Integration Building and testing every change automatically.

CI/CD

Continuous Integration / Continuous Delivery The automated build, test, scan and deploy pipeline.

CSP

Content Security Policy A browser header that limits which scripts and resources a page may load.

CVE

Common Vulnerabilities and Exposures The ID of one specific, publicly known vulnerability, such as CVE-2021-44228 (Log4Shell).

CVSS

Common Vulnerability Scoring System A 0–10 severity score: how bad a vulnerability would be if exploited.

CWE

Common Weakness Enumeration A catalogue of weakness types. CWE-89 is SQL injection; CWE-502 is unsafe deserialization.

ECR

Elastic Container Registry AWS's container image registry.

ECS

Elastic Container Service AWS's container service. Fargate is its serverless mode.

EOL

End of Life A version that no longer gets security patches.

EPSS

Exploit Prediction Scoring System The probability (0–1) that a CVE is exploited in the next 30 days. Published by FIRST.

FIRST

Forum of Incident Response and Security Teams The global body that maintains EPSS and CVSS.

GCP

Google Cloud Platform Google's cloud. GitHubScanner and GraphRAG run on it.

GCS

Google Cloud Storage GCP's object storage (buckets).

GPU

Graphics Processing Unit The accelerator used to host models. OCI "GPU shapes" are GPU-equipped VM sizes.

IaC

Infrastructure as Code Cloud resources defined in code and versioned, such as Terraform or CloudFormation.

IAM

Identity and Access Management Who can do what. "Cloud Run IAM" means only authenticated callers can reach the service.

IAP

Identity-Aware Proxy Google's SSO gate in front of a web app.

JWT

JSON Web Token A signed token that carries identity claims.

KMS

Key Management Service Managed encryption keys (AWS KMS, Cloud KMS).

LLM

Large Language Model A model such as Gemini or Claude that reads and writes text. Every agent here is an LLM plus tools.

OCI

Oracle Cloud Infrastructure Oracle's cloud. The service maps show the OCI equivalent of each component.

OCIR

OCI Registry OCI's container image registry.

OIDC

OpenID Connect An identity layer on OAuth 2.0. It lets a CI job prove who it is and get short-lived cloud credentials without stored keys.

OKE

Oracle Kubernetes Engine OCI's managed Kubernetes (formerly Container Engine for Kubernetes).

OSS

Open-Source Software Third-party packages. The "OSS licence + vuln" gate checks their licences and known CVEs.

OSV

Open Source Vulnerabilities Google's vulnerability database (osv.dev), keyed by package and version range.

OWASP

Open Worldwide Application Security Project The nonprofit behind the Top 10 risk lists, including the Top 10 for LLM Applications (LLM01 to LLM10).

PR

Pull Request A proposed code change, reviewed before it is merged.

QPM

Queries Per Minute A rate limit unit.

RDS

Relational Database Service AWS's managed SQL databases (PostgreSQL, MySQL and others).

RS256

RSA Signature with SHA-256 An asymmetric JWT signing algorithm. GitHub App JWTs use it.

S3

Simple Storage Service AWS's object storage.

SA

Service Account A non-human identity that a workload runs as.

SAST

Static Application Security Testing Scanning source code for vulnerabilities without running it.

SBOM

Software Bill of Materials An inventory of every dependency and version in a build.

SHA

Secure Hash Algorithm A "git SHA" is the hash that uniquely identifies a commit.

SHA-1

Secure Hash Algorithm 1 A 160-bit hash. Used here for deduplication and snippet-integrity checks, not for security.

SQL

Structured Query Language The standard language for querying relational databases.

SQLi

SQL injection Untrusted input that changes the meaning of a database query.

SQS

Simple Queue Service AWS's managed message queue.

STRIDE

Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege Microsoft's threat-modelling checklist.

STS

Security Token Service Exchanges one credential for a short-lived one. GCP STS powers WIF; AWS STS powers AssumeRole.

TTL

Time To Live How long a record or cache entry is kept before it expires.

VPC

Virtual Private Cloud A private network in the cloud. VPC Service Controls put a perimeter around GCP APIs to stop data exfiltration.

WIF

Workload Identity Federation Lets an outside identity (an AWS role, a GitHub OIDC token) act as a GCP service account without a JSON key.

XXE

XML External Entity An injection where an XML parser is tricked into reading local files or calling URLs.