GraphRAG vs Vector RAG

Overview

The Data Science group at CVS Health / Aetna wanted to know whether a knowledge graph beats plain vector search for questions about its data assets: file layouts, field definitions and production schedules. A question like "which feeds carry the claim-status field, and who owns them?" needs relationships, not just similar text.

I built both systems on the same corpus of 43 internal reference documents: 17 XLS, 15 DOC, 9 PDF, 1 DOCX and 1 XLSX. The corpus contained no member or PHI data.

• Vector RAG: a LangGraph pipeline over ChromaDB with Vertex AI embeddings.
• GraphRAG: Gemini extracts entities and relationships under a constrained schema into Neo4j. Answers are generated from graph facts.
• LLMOps layer: prompt registry, evaluator and SLA monitoring, plus an A/B harness that compares Gemini models.
• Deployment: both run on GCP Cloud Run through GitHub Actions. Moving to scale-to-zero and removing an always-on VM cut hosting cost by about 98%.

System

• Retrieval uses parameterized Cypher templates selected by intent, not free-form text-to-Cypher. Letting an LLM write queries against a database is a reliability risk and an injection risk.
  • The extension path is text-to-Cypher behind a read-only database user plus schema validation.
• Graph build: 51 LLM extraction calls in 9.7 minutes produced 1,943 nodes and 4,081 relationships.

GraphRAG or vector RAG?

GCP deployment

The cost story: about 98% lower

Lesson: scale-to-zero compute buys you nothing if the database is always on. The VM was the cost, not Cloud Run. Put state on a managed tier that also scales down, prune the registry, and initialize lazily.

• This was a pilot-scale workload, so the dollar amounts were small. The pattern applies to any bursty service.
• Cold starts. min-instances 0, lazy component init, and NLTK data baked into the image. For a latency SLA, set min-instances 1 and turn on startup CPU boost.

Model A/B results

The A/B harness compared Gemini models on the GraphRAG service, measuring success rate and latency:

Every failure was a client read timeout (30/60 s), not a model error. For interactive use, the "smarter" model lost, so 2.0 Flash stayed in production.

Engineering lessons

1. Aura → self-hosted VM → back to Aura. Self-hosting looked cheaper and wasn't. Deleting the VM required repointing the Cloud Run secrets and env vars to Aura. State placement drives cost.
2. Model and region availability. Gemini 2.5 wasn't available in us-east1, which led to a model fallback chain (a 404 falls back to 2.0 Flash).
3. ChromaDB on Cloud Run:
   • The collection's default embedding function didn't match the Vertex vectors, so queries pass precomputed 768-d query_embeddings.
   • Chroma rejects list metadata, so lists are JSON-serialized.
   • The DB is baked into the image, so the container is immutable and needs no volume.
4. LLM-as-judge pitfall. The evaluator kept returning the example scores from its own prompt. An LLM judge needs calibration against human-labeled examples, and an alarm when its scores show zero variance.

From pilot to production

• CI auth: keyless Workload Identity Federation everywhere.
• Access: IAM-authenticated Cloud Run or IAP behind SSO, with CORS restricted.
• Identity: a dedicated least-privilege service account (aiplatform.user, secretmanager.secretAccessor).
• Serving: gunicorn or uvicorn workers, not a development server.
• Chat memory: Memorystore (Redis) or Firestore instead of in-process state.
• Secrets: --set-secrets in every script and workflow.
• Before any real PHI: DLP / redaction at ingest, VPC Service Controls, CMEK, and a BAA-covered model endpoint.

The same system on AWS

Service map: GCP → AWS → OCI

What's the AWS equivalent of Cloud Run? There's no exact twin.

• Fargate is the workhorse, but it doesn't scale to zero on requests.
• App Runner is the closest developer experience.
• Lambda with a container image and the Web Adapter gives true scale-to-zero, with a 15-minute limit.
• On OCI, use Functions for scale-to-zero, or OKE with KEDA for Cloud Run-style semantics on Kubernetes.

Acronyms

Underlined acronyms show these definitions on hover or tap. Full glossary →

ALB

Application Load Balancer AWS's HTTP load balancer. It can enforce OIDC login itself.

API

Application Programming Interface How programs call each other, such as a REST endpoint.

APM

Application Performance Monitoring Request tracing and latency monitoring.

AWS

Amazon Web Services Amazon's cloud. GitHubScanner's CI/CD pipeline runs in AWS.

BAA

Business Associate Agreement The HIPAA contract a vendor (for example, a model endpoint) must sign before it can process PHI.

CI

Continuous Integration Building and testing every change automatically.

CI/CD

Continuous Integration / Continuous Delivery The automated build, test, scan and deploy pipeline.

CMEK

Customer-Managed Encryption Keys You hold the keys that encrypt your cloud data, and can revoke them.

CORS

Cross-Origin Resource Sharing The browser rule for which other websites may call your API.

DLP

Data Loss Prevention Finding and redacting sensitive data, for example before documents are ingested.

ECR

Elastic Container Registry AWS's container image registry.

ECS

Elastic Container Service AWS's container service. Fargate is its serverless mode.

GCP

Google Cloud Platform Google's cloud. GitHubScanner and GraphRAG run on it.

GCS

Google Cloud Storage GCP's object storage (buckets).

GenAI

Generative AI Models that generate text, code or images. "OCI GenAI" is Oracle's managed LLM service.

GPU

Graphics Processing Unit The accelerator used to host models. OCI "GPU shapes" are GPU-equipped VM sizes.

GraphRAG

Graph Retrieval-Augmented Generation RAG that retrieves facts from a knowledge graph (entities and their relationships) instead of text chunks.

HNSW

Hierarchical Navigable Small World A graph-based approximate nearest-neighbour index for fast vector search. ChromaDB uses it.

IAM

Identity and Access Management Who can do what. "Cloud Run IAM" means only authenticated callers can reach the service.

IAP

Identity-Aware Proxy Google's SSO gate in front of a web app.

IdP

Identity Provider The service behind SSO, such as Okta or Microsoft Entra ID.

KB

Knowledge Base "Bedrock KB" is Amazon Bedrock Knowledge Bases, a managed RAG service.

KEDA

Kubernetes Event-Driven Autoscaling Scales Kubernetes workloads on events, including down to zero.

KMS

Key Management Service Managed encryption keys (AWS KMS, Cloud KMS).

LLM

Large Language Model A model such as Gemini or Claude that reads and writes text. Every agent here is an LLM plus tools.

LLMOps

LLM Operations Running LLM apps in production: prompt versioning, evaluation, latency and success-rate monitoring.

MD5

Message Digest 5 A fast hash. Used here only to detect which files changed, not for security.

NLTK

Natural Language Toolkit A Python NLP library. Its data files are baked into the container image to cut cold-start time.

OCI

Oracle Cloud Infrastructure Oracle's cloud. The service maps show the OCI equivalent of each component.

OCIR

OCI Registry OCI's container image registry.

OIDC

OpenID Connect An identity layer on OAuth 2.0. It lets a CI job prove who it is and get short-lived cloud credentials without stored keys.

OKE

Oracle Kubernetes Engine OCI's managed Kubernetes (formerly Container Engine for Kubernetes).

PHI

Protected Health Information Health data covered by the US HIPAA rules. None was in the GraphRAG corpus.

PII

Personally Identifiable Information Data that identifies a person, such as a name, email or SSN.

RAG

Retrieval-Augmented Generation Look up relevant documents first, then have the LLM answer from them instead of from memory.

S3

Simple Storage Service AWS's object storage.

SHA

Secure Hash Algorithm A "git SHA" is the hash that uniquely identifies a commit.

SLA

Service-Level Agreement A promised target, such as a 12 s latency budget at 99% success.

SQL/PGQ

SQL Property Graph Queries The ISO SQL extension for graph queries. Oracle Database 23ai uses it for property graphs.

SSO

Single Sign-On One corporate login for many apps.

TLS

Transport Layer Security Encryption in transit: https://, neo4j+s://.

vCPU

Virtual CPU One virtual processor core allotted to a container or VM.

VM

Virtual Machine A rented server. GraphRAG's always-on Neo4j VM was most of its cost.

VPC

Virtual Private Cloud A private network in the cloud. VPC Service Controls put a perimeter around GCP APIs to stop data exfiltration.

WAF

Web Application Firewall Filters malicious HTTP traffic before it reaches the app.