Skip to main content
A batch is done when its status is completed, cancelled, or failed. Then you can page through its records as JSON or download them as files. Result files are deleted 180 days after the batch finishes.

Check status

GET /v1/batch/{batch_id} returns the batch. Poll every 10 to 30 seconds while status is queued, running, or cancelling, or let a webhook tell you when it’s done. A finished batch (trimmed):
A crawl can finish below its maxUrls when it runs out of pages in scope.

Read results

GET /v1/batch/{batch_id}/results pages through a finished batch’s records as JSON. It returns 409 BATCH_NOT_COMPLETED until the batch is final.
  • limit is 1–100 and defaults to 25. A page can close early to stay under about 8 MB, so keep requesting with cursor set to next_cursor while has_more is true. An unknown cursor returns 400.
  • Records aren’t in submission order. Match them with itemId; meta comes back too.
  • A successful record has status: "ok", final_url, metadata, cache_metadata, and markdown or html. Markdown batches with includeHTML return both. ocr_pages appears when OCR recovered PDF pages.
  • A failed record has status: "error", error_code, and message:

Download files

For large batches, download the files in results.files instead of paging. Each file is gzipped NDJSON, one record per line, with up to 50 records, so a 25,000-URL batch has about 500 files. File order carries no meaning. Each link expires 24 hours after the call that returned it (results.expires_at). Retrieve the batch again for new links. The data.batch.results links in a batch.* webhook expire 24 hours after the event, and a replay resends those same links. Result files are deleted 180 days after the batch finishes. After that, the links stop working and /results can’t return the records, so copy what you need within 180 days.

List batches

GET /v1/batch/list returns your batches, newest first. It returns 25 per page by default and up to 100 with limit. Filter with these parameters: Page with cursor set to next_cursor. A search with q or tags can return 503 SEARCH_UNAVAILABLE; retry it, or list without them.