GearNest

Scraping 8 stores concurrently, normalising 50k products, chunking 3M review texts, and generating vector embeddings for all of them is CPU and I/O bound. Rust’s async model (Tokio) runs all 8 store scrapers concurrently with zero data races guaranteed at compile time. Rayon handles CPU-bound normalisation in parallel. The memory footprint is roughly 10× lower than a JVM equivalent, which matters on constrained Cloud Run batch-job machines.

The pipeline is built around a StoreCrawler trait — each store’s implementation is transport-agnostic (clean HTTP, proxy rotation, or headless browser via chromiumoxide). Upgrading one store’s anti-bot approach is a one-file change that doesn’t touch the normaliser, chunker, or embedder.

"MSR PocketRocket 2" and "Mountain Safety Research Pocket Rocket II Stove" are the same product. Naive string similarity breaks immediately on real retailer title data. GearNest uses a three-tier resolution system: structural identifiers (GTIN/ASIN, O(1) lookup from affiliate APIs), structured attribute extraction (a ~40-brand alias table + model-number regex), and embedding similarity on canonical attribute strings as a fallback.

Each match gets a confidence tier (EXACT / HIGH / MEDIUM / CANDIDATE). CANDIDATE rows are written to the DB but filtered from the user-facing price comparison table — they sit in a review queue rather than silently corrupting the canonical product graph. Documented as ADR-007, the single most important invariant in the ingestion pipeline.

The chat is always product-scoped. Users ask about a specific product ("Is this good for below-freezing conditions?") and the system retrieves context from that product’s indexed chunks only. Because the filter is so selective (~75 rows per product), an HNSW vector index would degrade to a sequential scan — exact KNN over 75 embeddings is microseconds and 100% accurate. The pgvector indexes are btree on product_id, not HNSW.

Retrieval is stratified to prevent sentiment bias: semantic top-5 (with MMR for diversity) + top-3 negative review chunks (guaranteed failure modes regardless of query direction) + top-2 spec chunks. A pure top-10 cosine query on "Is this good for rain?" would cluster around rain chunks and miss the snow failure. Stratification makes answers trustworthy.

Session budget (5 questions per 2-hour session) uses a Redis WATCH/MULTI reserve-then-commit pattern — budget rolls back if HuggingFace never responds, preventing wasted questions on timeouts.

Cloud Run for the API (scale-to-zero, paid only on request) + Cloud SQL for Postgres (because Cloud Run can’t host pgvector) + Upstash Redis (serverless) + Vercel for the frontend. A Cloud Billing budget at $30 fires a Pub/Sub notification that triggers a Cloud Function — which suspends Cloud Run, downsizes Cloud SQL, and stops the pipeline scheduler. So the upper bound is enforced infrastructurally, not by trust.

The monorepo is deliberately split into three services (pipeline / api / web) each with its own CLAUDE.md scoping the agent to that service. Session 0 owns the contract — Postgres schema, OpenAPI spec, ADRs, infra. Sessions A/B/C each implement one service against the contract, never editing each other’s code. This makes parallel CLI work possible without merge conflicts and forces every cross-service question to surface as an OpenAPI or schema change.