POST /v1/map — discover URLs fast | Stekpad API

POST /v1/map

Discover all the URLs on a site without fetching the bodies. One credit per thousand URLs returned.

Get your API keyCost · 1 credit per 1,000 URLs

Overview

What this verb does.

map walks /robots.txt, every sitemap referenced there, sitemap indexes recursively, and optionally a shallow link crawl. It returns the URLs only, with minimal metadata. It does not render pages or transfer bodies — it is the fast, cheap, pre-step before a targeted crawl.

A common pattern: map → filter → crawl. Run map first to find every product URL, drop the ones you do not want, and pass the survivors to crawl.

Request

Make the call.

Pick your language. Every snippet is a real, runnable example.

curl -X POST https://api.stekpad.com/v1/map \
  -H "Authorization: Bearer stkpd_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "sources": ["sitemap", "robots"],
    "include_paths": ["/product/**"],
    "max_urls": 10000
  }'

Parameters

Request body.

Name	Type	Required	Description
url	string	required	Seed URL or domain.
sources	string[]	optional	Any of sitemap, robots, links. Defaults to ["sitemap", "robots"].
include_paths	string[]	optional	Glob filter applied to the result.
exclude_paths	string[]	optional	Glob filter applied to the result.
max_urls	int	optional	Hard cap. Default 50,000.
follow_sitemap_index	boolean	optional	Walk sitemap indexes recursively. Default true.

Response

What you get back.

map-response.json

json

{
  "run_id": "run_01HZ...",
  "urls": [
    { "url": "https://example.com/product/1", "source": "sitemap", "last_modified": "2026-04-01" },
    { "url": "https://example.com/product/2", "source": "sitemap" }
  ],
  "total": 4231,
  "credits_charged": 5
}

Errors

Structured, typed, actionable.

Code	When
no_sitemap_found	The site has neither a sitemap nor a usable robots.txt and links was not in sources.
sitemap_too_large	A single sitemap exceeded max_urls and follow_sitemap_index was false.
target_unreachable	The robots.txt request failed.

Every error includes a code, a human message, and a guidance field with the exact action to take.

The other four verbs.

scrape1 credit

Fetch one URL. Markdown, JSON, HTML or a screenshot.

POST /v1/scrape
{ "url": "https://example.com" }

crawl1 / page

Walk an entire site. Async, with webhooks and a run_id you can poll.

POST /v1/crawl
{ "url": "https://example.com" }

extract5 / URL

JSON Schema in, validated structured fields out.

POST /v1/extract
{ "urls": ["https://acme.com/about"], "schema": {...} }

search5 + 1 / result

Web search and scrape, same call. Built for AI agents.

POST /v1/search
{ "query": "best RAG frameworks", "scrape_results": true }

Try /v1/map now.

Get an API key, paste the curl, watch the row land in your dataset.

Get your API key Read the docs

Name

Type

Required

Description

url

string

required

Seed URL or domain.

sources

string[]

optional

Any of sitemap, robots, links. Defaults to ["sitemap", "robots"].

include_paths

string[]

optional

Glob filter applied to the result.

exclude_paths

string[]

optional

Glob filter applied to the result.

max_urls

int

optional

Hard cap. Default 50,000.

follow_sitemap_index

boolean

optional

Walk sitemap indexes recursively. Default true.

Code

When

no_sitemap_found

The site has neither a sitemap nor a usable robots.txt and links was not in sources.

sitemap_too_large

A single sitemap exceeded max_urls and follow_sitemap_index was false.

target_unreachable

The robots.txt request failed.