In one breath
- Question (CVS Health / Aetna Data Science): does a knowledge graph beat vector search for questions about data assets: file layouts, field definitions, production schedules?
- Built both on the same 43 reference documents (no member or PHI data): a LangGraph vector RAG and a Neo4j GraphRAG.
- LLMOps layer: prompt registry, evaluator, SLA monitoring, model A/B harness.
- Deployed on Cloud Run via GitHub Actions. Scale-to-zero plus deleting an always-on VM cut hosting cost ~98%.
Key numbers
| Corpus | 43 docs: 17 XLS · 15 DOC · 9 PDF · 1 DOCX · 1 XLSX |
| Graph build | 51 LLM calls · 9.7 min |
| Graph | 1,943 nodes · 4,081 relationships |
| Schema allowlist | 35 labels × 33 relationship types |
| Vector store | ChromaDB · 910 chunks · 768-d · HNSW |
| Chunking | RAG 1,500 / 200 overlap · graph loader 4,000 / 200 |
| SLA monitor | 12 s latency · 99% success |
| Hosting | ≈ $7.38/mo → ≈ $0.15/mo |
GraphRAG pipeline
- Ingest: one Document per sheet or file (columns, row count, 3 sample rows) → LangChain
LLMGraphTransformer(Gemini, temp 0.1, allowlisted schema) →MERGEin batches of 3 → Neo4j Aura. - Incremental: MD5 per file, so only changed files are reprocessed; quota tracker, retries, 300 s timeout.
- Query: question + persona (Default · Technical · Business · Compliance) → intent keywords → parameterized Cypher templates → graph facts → Gemini 2.5 Flash (fallback 2.0 Flash) → answer, logged to LLMOps.
Vector RAG twin
- LangGraph
StateGraph, 7 nodes: analyze → plan → retrieve → filter/rank → build context → generate → validate. - ChromaDB embedded and baked into the image: immutable container, no volume.
- Vertex
text-embedding-005, 768-d.
Why templates, not text-to-Cypher
- Letting an LLM write database queries is a reliability risk and an injection risk.
- Intent-matched, parameterized templates are predictable and testable.
- Extension path: text-to-Cypher behind a read-only DB user plus schema validation.
Vector RAG vs GraphRAG
| Vector RAG | GraphRAG | |
|---|---|---|
| Best for | local, single-passage | multi-hop, relational |
| Ingest cost | 1 embedding per chunk | LLM extraction per chunk |
| Breaks when | chunking / embedding is off | the schema is wrong |
| Explains with | retrieved passages | the node-and-edge path |
GCP deployment
- CI: GitHub Actions →
docker build --platform linux/amd64, tag = commit SHA → Artifact Registry. - Cloud Run: 1 vCPU · 2 GiB · min 0 / max 10 · timeout 300 s · concurrency 80. RAG service: gunicorn 1 worker × 8 threads.
- Secrets: Neo4j URI, user, password via Secret Manager (
--set-secrets). - Neo4j Aura over
neo4j+s://(Bolt over TLS), so no VPC connector. - Post-deploy
/healthcheck, 5 retries. - CI auth: service-account key (GraphRAG) vs keyless WIF (vector RAG). Target: WIF everywhere.
The cost story
| Before ≈ $7.38/mo | After ≈ $0.15/mo |
|---|---|
| Self-hosted Neo4j on an always-on n1-standard-2 VM | Neo4j Aura (managed) |
| ~41 GB of old images | keep the last 2 images |
| Cloud Run | min-instances 0 + lazy init |
Lesson: scale-to-zero compute buys nothing if the database is always on. The VM was the cost, not Cloud Run.
Cold starts: lazy init and NLTK data baked in. For a latency SLA: min-instances 1 + startup CPU boost.
Model A/B
| Model | Success | Median latency |
|---|---|---|
| gemini-2.0-flash-001 | 100% | 12.2 s |
| gemini-2.0-flash-exp | 96% | 13.8 s |
| gemini-2.5-flash-preview | 75% | 28.3 s |
| gemini-2.5-pro-preview | 32% | 47.2 s |
Every failure was a client read timeout (30/60 s), not a model error. The "smarter" model lost for interactive use, so 2.0 Flash stayed.
Engineering lessons
- Aura → self-hosted VM → back to Aura. Self-hosting looked cheaper and wasn't. State placement drives cost.
- Region availability: Gemini 2.5 wasn't in us-east1 → fallback chain (404 → 2.0 Flash).
- ChromaDB: default embedding function didn't match Vertex vectors → pass precomputed
query_embeddings; list metadata JSON-serialized. - LLM-as-judge echoed the example scores from its own prompt → calibrate against human labels, alarm on zero variance.
Pilot → production
- Keyless WIF for all CI.
- IAM-authenticated Cloud Run or IAP behind SSO; CORS restricted.
- Least-privilege SA (
aiplatform.user,secretmanager.secretAccessor). - gunicorn / uvicorn, not a dev server.
- Chat memory in Memorystore (Redis) or Firestore.
- Before any PHI: DLP at ingest, VPC Service Controls, CMEK, a BAA-covered model endpoint.
Cloud Run on AWS?
There's no exact twin.
- Fargate is the workhorse but doesn't scale to zero on requests.
- App Runner is the closest developer experience.
- Lambda + container image + Web Adapter gives true scale-to-zero, with a 15-minute limit.
- OCI: Functions for scale-to-zero, or OKE + KEDA for Cloud Run-style semantics on Kubernetes.
Service map: GCP → AWS → OCI
| Component | GCP (built) | AWS | OCI |
|---|---|---|---|
| Serverless container | Cloud Run (min 0) | ECS Fargate / App Runner · Lambda + Web Adapter for true scale-to-zero | OCI Functions · Container Instances · OKE + KEDA |
| Registry | Artifact Registry | ECR | OCIR |
| CI auth | Workload Identity Federation | GitHub OIDC → IAM role | OCI DevOps · Identity Domains token exchange |
| Secrets | Secret Manager | Secrets Manager | OCI Vault |
| LLM | Vertex AI Gemini | Bedrock (Claude, Llama, Titan) | OCI Generative AI · self-hosted on GPU shapes |
| Embeddings | text-embedding-005 | Titan Text Embeddings v2 · Cohere Embed | OCI GenAI Cohere Embed |
| Graph DB | Neo4j Aura | Neptune (openCypher / Gremlin) · Neo4j Aura on AWS | Oracle Database 23ai property graphs (SQL/PGQ) |
| Vector store | ChromaDB (embedded) | OpenSearch Serverless · Aurora pgvector · Bedrock KB | Oracle 23ai AI Vector Search · PostgreSQL pgvector |
| Object storage | GCS | S3 | Object Storage |
| Edge auth | IAP / Cloud Run IAM | ALB OIDC · Cognito · API Gateway | API Gateway + Identity Domains · OCI WAF |
| Logs / metrics | Cloud Logging / Monitoring | CloudWatch · X-Ray | OCI Logging · Monitoring · APM |
Acronyms on this sheet
Underlined acronyms show these definitions on hover or tap. Full glossary →
- ALB
- Application Load Balancer AWS's HTTP load balancer. It can enforce OIDC login itself.
- API
- Application Programming Interface How programs call each other, such as a REST endpoint.
- APM
- Application Performance Monitoring Request tracing and latency monitoring.
- AWS
- Amazon Web Services Amazon's cloud. GitHubScanner's CI/CD pipeline runs in AWS.
- BAA
- Business Associate Agreement The HIPAA contract a vendor (for example, a model endpoint) must sign before it can process PHI.
- CI
- Continuous Integration Building and testing every change automatically.
- CMEK
- Customer-Managed Encryption Keys You hold the keys that encrypt your cloud data, and can revoke them.
- CORS
- Cross-Origin Resource Sharing The browser rule for which other websites may call your API.
- DLP
- Data Loss Prevention Finding and redacting sensitive data, for example before documents are ingested.
- ECR
- Elastic Container Registry AWS's container image registry.
- ECS
- Elastic Container Service AWS's container service. Fargate is its serverless mode.
- GCP
- Google Cloud Platform Google's cloud. GitHubScanner and GraphRAG run on it.
- GCS
- Google Cloud Storage GCP's object storage (buckets).
- GenAI
- Generative AI Models that generate text, code or images. "OCI GenAI" is Oracle's managed LLM service.
- GPU
- Graphics Processing Unit The accelerator used to host models. OCI "GPU shapes" are GPU-equipped VM sizes.
- GraphRAG
- Graph Retrieval-Augmented Generation RAG that retrieves facts from a knowledge graph (entities and their relationships) instead of text chunks.
- HNSW
- Hierarchical Navigable Small World A graph-based approximate nearest-neighbour index for fast vector search. ChromaDB uses it.
- IAM
- Identity and Access Management Who can do what. "Cloud Run IAM" means only authenticated callers can reach the service.
- IAP
- Identity-Aware Proxy Google's SSO gate in front of a web app.
- KB
- Knowledge Base "Bedrock KB" is Amazon Bedrock Knowledge Bases, a managed RAG service.
- KEDA
- Kubernetes Event-Driven Autoscaling Scales Kubernetes workloads on events, including down to zero.
- LLM
- Large Language Model A model such as Gemini or Claude that reads and writes text. Every agent here is an LLM plus tools.
- LLMOps
- LLM Operations Running LLM apps in production: prompt versioning, evaluation, latency and success-rate monitoring.
- MD5
- Message Digest 5 A fast hash. Used here only to detect which files changed, not for security.
- NLTK
- Natural Language Toolkit A Python NLP library. Its data files are baked into the container image to cut cold-start time.
- OCI
- Oracle Cloud Infrastructure Oracle's cloud. The service maps show the OCI equivalent of each component.
- OCIR
- OCI Registry OCI's container image registry.
- OIDC
- OpenID Connect An identity layer on OAuth 2.0. It lets a CI job prove who it is and get short-lived cloud credentials without stored keys.
- OKE
- Oracle Kubernetes Engine OCI's managed Kubernetes (formerly Container Engine for Kubernetes).
- PHI
- Protected Health Information Health data covered by the US HIPAA rules. None was in the GraphRAG corpus.
- RAG
- Retrieval-Augmented Generation Look up relevant documents first, then have the LLM answer from them instead of from memory.
- S3
- Simple Storage Service AWS's object storage.
- SA
- Service Account A non-human identity that a workload runs as.
- SHA
- Secure Hash Algorithm A "git SHA" is the hash that uniquely identifies a commit.
- SLA
- Service-Level Agreement A promised target, such as a 12 s latency budget at 99% success.
- SQL/PGQ
- SQL Property Graph Queries The ISO SQL extension for graph queries. Oracle Database 23ai uses it for property graphs.
- SSO
- Single Sign-On One corporate login for many apps.
- TLS
- Transport Layer Security Encryption in transit:
https://,neo4j+s://. - vCPU
- Virtual CPU One virtual processor core allotted to a container or VM.
- VM
- Virtual Machine A rented server. GraphRAG's always-on Neo4j VM was most of its cost.
- VPC
- Virtual Private Cloud A private network in the cloud. VPC Service Controls put a perimeter around GCP APIs to stop data exfiltration.
- WAF
- Web Application Firewall Filters malicious HTTP traffic before it reaches the app.
- WIF
- Workload Identity Federation Lets an outside identity (an AWS role, a GitHub OIDC token) act as a GCP service account without a JSON key.
More quick reference