GitHubScanner

In one breath

• AI security reviewer for repositories, on Google ADK + Gemini.
• Specialist agents find issues. Deterministic validators check every finding before a human sees it.
• Fixes are generated in a throwaway clone and exported as patches. Nothing is pushed.
• Runs on Cloud Run, deployed from AWS CodePipeline through keyless Workload Identity Federation.

Key numbers

Scan flow

1. POST /scan → root agent.
2. Shallow clone with a GitHub App token, detect the stack.
3. Planner → Security agent (regex hints → LLM judgment) and Dependency agent (manifests and lockfiles → OSV, EPSS, EOL, licences, typosquats).
4. Hallucination validator → report agent.
5. Grade, HTML/JSON report to GCS, signed URL back.
6. Remediation on demand: temp clone → patch → developer review.

Grade: 100 − 25·C − 15·H − 8·M − 3·L − 1·I → A–F (Critical, High, Medium, Low, Info).

CVE enrichment

• package@version → SQLite CVE cache (7-day TTL) → OSV.dev (breaker: 3 fails) → EPSS (breaker: 5 fails).
• EPSS bands: > 0.7 critical · > 0.3 high · > 0.1 medium · > 0.01 low.
• Prioritize by severity rank, then exploitability.
• CVSS = how bad. EPSS = how likely in the next 30 days. CVSS 9.8 with EPSS 0.001 can wait; CVSS 7.5 with EPSS 0.9 is today's problem.

Trust pipeline (5 layers)

1. Grounding: cite file + line + snippet, 8-item pre-output checklist, ID prefixes (JAVA-, PY-, JS-, DEP-, SEC-), max 20 findings.
2. Regex hints: the model judges hints instead of inventing from nothing.
3. HallucinationValidator (no LLM judge): approved CWE list · severity ↔ exploitability · line-number sanity · snippet SHA-1 · 22 "synthetic evidence" phrases, ≥ 2 indicators = flagged.
4. Gate: > 5% invalid → strip invalid findings and flag the scan.
5. Consensus (opt-in): N parallel scans, keep findings in ≥ 67% of runs. Verification mode on re-scan: does the exact snippet or hash still exist? 3-level fingerprint; unsure ⇒ still open.

Why no LLM judge

• A judge model shares the generator's failure modes and doubles the cost.
• String and hash checks are cheap, reproducible and auditable.
• The 5% is a gate, not an accuracy claim.
• Next: assert the cited snippet appears within ±N lines of the claimed line.

Secrets and identity

• GitHub App, not a personal access token: short-lived, repo-scoped.
• App private key in Secret Manager.
• RS256 JWT (cached 9 min) → installation token (cached 1 h).
• No long-lived credential in a repo or a log line.

Keyless AWS → GCP deploy

1. CodePipeline (dev / qa) → CodeBuild (Python 3.12, hardened base image).
2. Security gates: SAST · OSS licence + vuln · image scan · code quality.
3. Task-role credentials → GCP STS token exchange (aws4_request) → impersonate one deploy SA.
4. Push to Artifact Registry, tagged latest, vX.Y.Z and git SHA.
5. Terraform-managed Cloud Run: non-root, port 8080, /healthcheck.

No JSON service-account keys anywhere, so nothing to leak or rotate.

Cloud design in the app

• Env detection: K_SERVICE set → config from Secret Manager, else .env.
• Scan state: SQLite synced to GCS (startup, every 5 min, after each scan, shutdown) → forces max-instances=1. Scale-out path: Cloud SQL Postgres / AlloyDB or Firestore.
• Long scans: async. POST /scan returns scan_id → Cloud Tasks / Pub/Sub → Cloud Run Job → client polls or gets a webhook.
• Region pinning during a scan: Gemini availability differs by region.

Scaling out

• GitHub webhook (push / PR) → queue → scan workers → Postgres.
• Shared CVE cache removes the single-instance constraint.
• Dedup by commit SHA; re-scan only changed files + the dependency diff.
• SAST + LLM, not either/or: SAST is precise on known patterns; the LLM adds context (is this permitAll on a sensitive route? do two mediums chain into a critical?).

Engineering lessons

1. LLM "fixes" overwrote whole files with a snippet → size guards, read-only by default. Validate LLM output like user input.
2. Non-deterministic re-scans couldn't answer "is it fixed?" → verification mode with deterministic IDs and fingerprints.
3. Gemini 3 looped at temperature < 1.0 → pinned to 1.0, thinking budget moved to the planner.
4. Network controls blocked Vertex AI; proxies broke HTTP/2 → error classifier reports infrastructure errors, not "no findings".

Testing an LLM agent

• conftest.py mocks google.adk globally.
• Tests cover tools, parsers, validators, grading math and prompt content (the grounding rules must be in the prompt).
• Canned JSON fixtures stand in for model output, including malformed output.
• End-to-end runs against WebGoat and Spring Boot 2.6.