Configuring crawl rules and exclusions

Crawl rules let you tune how aggressively we crawl, what we skip, and how we authenticate. Settings live in two places: Settings → Crawl (workspace defaults) and the per-audit Advanced panel (one-off overrides).

Crawl depth

Maximum number of link hops from the seed URL. Default is 5.

1	Just the seed URL. Useful for spot-checking a single page.
2-3	Quick overview of a small site. Covers most marketing sites.
4-5	Full coverage of typical sites under 5k pages.
6-10	Deep coverage. Use when you have orphan sections accessed via deep navigation.
unlimited	Crawl until exhausted. Combined with a max-pages cap recommended for very large sites.

Concurrency and delay

Concurrency controls how many requests we have in flight at once. Delay adds a pause between requests per worker. Both directly affect crawl speed and origin load.

crawl_max_concurrent	Default 3. Range 1-10. Multi-tenant shared hosts may need 1; high-traffic Cloudflare sites can handle 10.
crawl_delay_ms	Default 500. Range 0-10000. Higher delays for fragile origins.

URL exclusion patterns

Comma-separated list of substrings or regex patterns to skip. Plain strings = substring match; wrap in slashes = regex.

text
/wp-admin, /cart, /checkout, ?orderby=, /\/page\/[0-9]+\/?$/

Excluded URLs are still discovered (so we can detect orphan-link issues) but not fetched, rendered or graded.

Robots.txt

On by default. We obey Disallow/Allow rules matching our user agent. Disable for diagnostic audits where you want to see everything regardless of robots.txt:

text
crawl_respect_robots = false

External links

Off by default. When enabled, we follow links to other domains for one level. Useful for audit-style "what does my outbound link profile look like" reports.

Authentication

Two methods supported, configured under Settings → Crawl → Authentication:

HTTP basic auth

text
crawl_auth_username = admin
crawl_auth_password = ...

Session cookie

Paste the full cookie string (everything after Cookie: in a captured request):

text
crawl_auth_cookie = session_id=abc123; auth_token=xyz...

Per-audit override

Audits → New audit → Advanced lets you override depth, exclusion patterns and seed URLs for a single run. Workspace defaults aren't affected.

API equivalent

All crawl rules can be set when starting an audit via API:

bash
curl -X POST https://api.semoptimiser.com/v1/audits \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "depth": 5,
    "max_concurrent": 5,
    "delay_ms": 200,
    "excludes": ["/wp-admin", "/cart"]
  }'

Crawl depth

Maximum number of link hops from the seed URL. Default is 5.

1	Just the seed URL. Useful for spot-checking a single page.
2-3	Quick overview of a small site. Covers most marketing sites.
4-5	Full coverage of typical sites under 5k pages.
6-10	Deep coverage. Use when you have orphan sections accessed via deep navigation.
unlimited	Crawl until exhausted. Combined with a max-pages cap recommended for very large sites.

Concurrency and delay

Concurrency controls how many requests we have in flight at once. Delay adds a pause between requests per worker. Both directly affect crawl speed and origin load.

crawl_max_concurrent	Default 3. Range 1-10. Multi-tenant shared hosts may need 1; high-traffic Cloudflare sites can handle 10.
crawl_delay_ms	Default 500. Range 0-10000. Higher delays for fragile origins.

Authentication

Two methods supported, configured under Settings → Crawl → Authentication:

HTTP basic auth

text

crawl_auth_username = admin crawl_auth_password = ...

Session cookie

Paste the full cookie string (everything after Cookie: in a captured request):

text

crawl_auth_cookie = session_id=abc123; auth_token=xyz...

API equivalent

All crawl rules can be set when starting an audit via API:

bash

curl -X POST https://api.semoptimiser.com/v1/audits \ -H "Authorization: Bearer sk_live_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "depth": 5, "max_concurrent": 5, "delay_ms": 200, "excludes": ["/wp-admin", "/cart"] }'

Configuring crawl rules and exclusions

Crawl depth

Concurrency and delay

URL exclusion patterns

Robots.txt

External links

Authentication

HTTP basic auth

Session cookie

Per-audit override

API equivalent

Audit, index and rank –
without a fifth subscription.

Configuring crawl rules and exclusions

Crawl depth

Concurrency and delay

URL exclusion patterns

Robots.txt

External links

Authentication

HTTP basic auth

Session cookie

Per-audit override

API equivalent

Audit, index and rank –
without a fifth subscription.

Configuring crawl rules and exclusions

Crawl depth

Concurrency and delay

URL exclusion patterns

Robots.txt

External links

Authentication

HTTP basic auth

Session cookie

Per-audit override

API equivalent

Audit, index and rank –without a fifth subscription.

Configuring crawl rules and exclusions

Crawl depth

Concurrency and delay

URL exclusion patterns

Robots.txt

External links

Authentication

HTTP basic auth

Session cookie

Per-audit override

API equivalent

Audit, index and rank –without a fifth subscription.

Audit, index and rank –
without a fifth subscription.

Audit, index and rank –
without a fifth subscription.