Price Tracker: Diagnosing a Live Scraping Pipeline Against Adversarial, Moving Targets
A Python-based price monitoring system tracking Apple hardware and 5K monitors across eight Canadian retailers, built and maintained through iterative AI-assisted debugging rather than a single build session. Surfaces live prices to a Notion dashboard with full price history in SQLite, and flags new all-time lows automatically.
Why Price Tracking Retail Targets Is Harder Than It Looks: The Moving Target Problem
Search infrastructure changes without notice, bot detection escalates mid-session, and a scraper that works cleanly on day one can silently break – or worse, silently return wrong data – a week later with no obvious signal that anything changed. Most “AI-built” scrapers are framed around a single prompt-to-product session; this one had to be maintained against retailers actively working against it.
Ambiguous failures
"No price found" looks identical in the log whether a scraper broke or a product is genuinely sold out
Silent data gaps
A regex can extract data correctly for most products and silently return nothing for an entire category, with no error anywhere
Moving infrastructure
Retailers migrate search backends and escalate anti-bot defenses overnight, breaking scrapers with no warning
Product Strategy: A Diagnose-Fix-Verify Loop, Not a One-Shot Build
I scoped price-tracker as an evidence-driven maintenance system. Rather than treating Claude as a one-shot code generator, this project ran as a diagnose-fix-verify loop across multiple sessions: paste real Terminal output, get a root-cause read grounded in the actual log lines – not assumptions – get a scoped fix, re-run, repeat.
- Verify before recording – every scraper must confirm the seller before a price is ever written to the database; if it can’t confirm, it returns nothing rather than a guess.
- Classify before re-diagnosing – an OUT_OF_STOCK vs NO_MATCH distinction was added to the Apple Refurbished scraper after cross-referencing live Apple stock confirmed certain SKUs were genuinely sold out, not broken – so future empty results don’t trigger a fresh investigation from scratch.
- Decide with evidence, not sunk cost – retailers get formally paused or deprioritized only after a live investigation produces a clear failure pattern, documented with the reasoning, not abandoned quietly or chased indefinitely.
My Approach: Evidence-Based Debugging Against Live, Adversarial Targets
I built and maintain price-tracker using Python, Playwright, SQLite, and the Notion API, running via cron and debugged through Claude across live sessions grounded in real terminal output rather than guesswork.
- Reading production logs as evidence, not guessing at fixes – a run that should have taken 20 minutes took 98. Instead of patching timeout logic blind, the bug was traced through log timestamps to a specific unenforced deadline three layers deep in a retry loop.
- Distinguishing “broke” from “working as intended” – when Apple Refurbished listings for Mac Studio and Mac Mini stopped returning prices, cross-referencing Apple’s live refurb stock confirmed those SKUs were simply sold out, not a selector failure.
- Verifying claims against a live target before writing a fix – when the Best Buy CA integration was consistently 404ing, the response was to fetch Best Buy’s actual pages directly, outside the scraper, and confirm the REST endpoint didn’t exist at all while the category pages loaded cleanly.
- Making a build-vs-abandon call with evidence, not sunk cost – after repeated runs and a live investigation into Best Buy’s bot detection showed zero successful price checks and escalating multi-minute hangs, the retailer was formally paused – code retained, disabled by design, documented with the reasoning.
Current state:
- 27 product-retailer pairs tracked across 8 evaluated Canadian retailers
- Confirmed working: Apple Store CA, Apple Refurbished CA (tile-matching and stock classification verified; price extraction from matched tiles is the current in-progress item), and Newegg CA, cross-confirmed independently via ASUS Canada’s own listings
- Formally documented as blocked or deprioritized rather than left as unexplained silent failures: Best Buy CA and Amazon CA (paused after live bot-detection investigation), Memory Express and Staples (Cloudflare-blocked), Canada Computers (search path changed, currently next in the fix queue)
Scraping Engine
Playwright (headless Chromium) - per-retailer seller verification - OUT_OF_STOCK / NO_MATCH classification - compound Apple chip-matching logic (M4 Pro vs M4, M4 Max, etc.)
Data & History
SQLite (price_records + price_history) - lowest-price tracking per product - full run history retained for trend analysis
Dashboard & Sync
Notion API dashboard - planned RAM/storage spec extraction per SKU - cross-retailer price confirmation (ASUS Canada vs. Newegg)
Runtime & Ops
Python, Playwright, SQLite - Cursor Composer diagnose-fix loop driven by real terminal output - test_refurb.py isolated test harness to fast-track Apple Refurb debugging without full tracker runs
The Spotlight Feature: The GB-vs-TB Regex Gap: The Bug That Never Threw an Error
The Problem
A regex correctly extracted RAM and storage specs for every product configured in gigabytes - and silently returned nothing for every product configured in 1TB, with no exception, no log warning, no crash. The pipeline kept running. The data just quietly stopped being complete for an entire category of configurations.
The Execution
The gap wasn't caught by a stack trace, because there wasn't one. It surfaced by comparing a Notion dashboard screenshot against the underlying scrape log, line by line, and noticing that every 1TB SKU was missing its spec field while every GB SKU had one. That observation, not an error message, was the actual signal - which meant the fix started with re-deriving the unit-matching logic against both formats before touching a single line of the scraper.
The Win
Silent failures are the most expensive kind, because nothing tells you they happened. Catching this one from a visual diff rather than a log line reinforced a standing rule for the rest of the project: don't just check that a run completed without errors, check that the output is actually complete.
