Site Crawler for Links and Tables

live$0.006/pageEnrichment

Last updated November 6, 2026

Crawl a website from a start address and get every page's links, download links, HTML tables, and lists as structured rows. No code, no AI, no browser, no login. Deterministic output, robots.txt obeyed by default.

View on Apify Store Create an Apify account

Features

Internal links, external links, and download links by file type.
HTML tables as rows with their headers, and lists as text.
Follows links and the sitemap, and obeys robots.txt by default.
No browser and no AI, so the same page gives the same output.
Pay per event: Page row at $0.006.

Input

start_urlsarrayoptional

Web addresses to start from, up to 100. A bare domain such as example.com is read as https.

max_pagesstringoptional

Pages to read across the whole run, up to 10,000. Default 100.

max_depthstringoptional

How many links to follow from a start address. 0 reads only the start address. Default 10.

same_domain_onlystringoptional

yes (default): follow links only on the start address's host and its subdomains. no: also follow links to other sites, up to Maximum pages. Links to other sites are listed either way.

extractarrayoptional

Any of links, tables, lists, download_links. Default links, tables, lists. download_links adds one separate row per file link, charged as download-link-row; the download_links field on page rows is always filled at no extra charge.

download_extensionsarrayoptional

File extensions that count as download links. Default pdf, zip, xlsx, xls, docx, doc, pptx, csv, mp3, mp4, exe, dmg, msi, pkg, gz, tar, tgz, 7z, rar, iso, apk. Files are listed, never downloaded.

use_sitemapstringoptional

yes (default): also queue the addresses in /sitemap.xml, which finds pages no link reaches. no: follow links only.

obey_robotsstringoptional

yes (default): skip pages robots.txt disallows and honor its crawl delay up to 5 seconds. no: read them, only if you have the right to.

max_charge_usdstringoptional

Optional. The run stops before a charge that would pass this amount. Summary rows are always returned.

proxy_modestringoptional

pool (default): the Apify proxy pool only, charged as page-row; a page that refuses it gets a free blocked row. auto: a refused page is retried on residential addresses, and a page read that way is charged as residential-page-row. residential: every page uses residential addresses and is charged as residential-page-row.

Output schema

json

{
  "row_type": "string",
  "start_url": "string",
  "url": "string",
  "final_url": "string",
  "http_status": "integer",
  "depth": "integer",
  "found_from": "string",
  "title": "string",
  "internal_links": "array",
  "external_links": "array",
  "download_links": "array",
  "tables": "array",
  "lists": "array",
  "likely_needs_javascript": "boolean",
  "link_url": "string",
  "link_text": "string",
  "file_extension": "string",
  "link_scope": "string",
  "pages_requested": "integer",
  "pages_charged": "integer",
  "pages_failed": "integer",
  "pages_blocked": "integer",
  "pages_skipped_robots": "integer",
  "pages_from_sitemap": "integer",
  "pages_left_in_queue": "integer",
  "download_links_found": "integer",
  "row_status": "string",
  "error_reason": "string",
  "billable_events_json": "string",
  "scraped_at": "string"
}

Pricing

$0.006per page

Event	Free plan price
Page row	$0.006
Residential page row	$0.02
Download link row	$0.0001
Actor Start	$0.0055

Pay per event on the Apify platform. Paid Apify plans pay less per event. No subscription, no seat license.

On the Apify free plan the actor stops at $1.50 of charges a month, about 250 pages. Any paid Apify plan removes the cap.

Limits

No browser. A page built by JavaScript in the visitor's browser comes back with little or nothing, marked likely_needs_javascript true. Use a browser based crawler for those sites.
Sites behind an anti bot wall (for example a Cloudflare challenge) can refuse the proxy pool. They return free blocked rows; proxy_mode auto may get through at the residential price, and may not.
Forms, logins, and pages reached only through a site's search box are not crawled.
Limits per page row: 5,000 links in each link field, 50 tables, 2,000 rows a table (row_count gives the real count), and 100 lists of 500 items. A row near the Apify 9 MB limit drops lists first, then tables, then links, and says so in error_reason.
A run queues up to 5 times Maximum pages plus 1,000 addresses, at most 50,000.
Subdomains of the start host count as the same site.
A link is called a download by its file extension. A file served from an address with no extension is not listed as a download.

Available on

Apify Store