Why does a web scraper suddenly return empty fields?
Possible causes include changed page structure, different content or an unexpected notice page. A successful HTTP response does not establish successful data extraction. In my shop research tool I separated directory discovery from website enrichment. That separation helps investigate source changes, preserve useful records and retry the steps that actually failed.
Research checked: 2026-09-11 · Cover: AI-generated illustration
Observe transport and content separately
I would ask three separate questions: did the page load, does it match the expected page type, and are the required values plausibly present? A successfully loaded notice page may satisfy the first check while being useless for extraction.
Each important field needs a missing-value policy. An optional social link is different from the discovered shop’s domain. Several fields disappearing together suggest a different problem from one incomplete record.
Do not retry every failure indefinitely
Crawlee documents bounded retries and handlers for requests that ultimately fail. These mechanisms are useful when matched to the cause. A temporary network error may disappear on retry; a permanently changed page structure probably will not.
I would label unresolved enrichment attempts and preserve the original discovery. Access restrictions and blocks must be respected. The appropriate next step can be a maintenance task rather than increasing request volume against the source.
The decision at a glance
- 01FetchPage available and appropriate?
- 02ExtractionRequired fields plausible?
- 03ReviewInspect changes and preserve values
Detect changes with a small representative sample
For maintenance testing, I would use approved example documents covering relevant page types. A parser update must preserve still-valid cases while fixing the changed case. This reveals whether an improvement for one source harms another.
In production I would observe the proportion of missing required fields. A sudden increase prompts inspection of the source and extraction rules. Holding questionable new results for review is preferable to overwriting verified values with empty fields.
A scraper is a maintained data process
My Shop Directory Lead Scraper treats discovery, enrichment, deduplication and export as parts of one workflow. For a new project, I would establish who reviews source changes and how failures become visible.
If you need custom web scraping, bring sample sources, required fields and the intended use of the results. I can design a bounded pipeline with an understandable review view. The team should still know which records to trust after a source changes.
