Develofly Autopilot
Templates › Website, SEO & Monitoring › Broken-link crawler

Broken-link crawler

Crawl a site and report every broken link (404s) to a sheet.

Website, SEO & MonitoringReady to run on its sampleWeekly

Download the app, freeOpen in Autopilot

Broken-link crawler: what it does, step by step

What it does

Crawls your site from the start page (same site only, up to the page limit), finds every link that answers 404, 410, a server error or can't be reached, and lists each one with the page it's on. It can also check links to other sites and images. Results open in Excel; with alerts on you're told when something breaks.

How it works

  1. Opens the page
  2. Picks out the values
  3. Goes through each item
  4. Adds rows to your sheet
  5. Decides what to do next
  6. Calls the service's API
  7. Alerts you
  8. Sends a chat message
All 17 steps (as they appear in the app)
  1. Open {{start_url}}
  2. Run JavaScript → _crawl
  3. Read values: pages_checked, pages_not_checked, _broken, _external
  4. broken_links = 0
  5. For each bad in {{_broken}} (2 steps)
  6. broken_links = {{broken_links|0}} + 1
  7. Add row: link, status, found_on, kind
  8. If check_external equals yes
  9. For each ext in {{_external}} (2 steps)
  10. GET {{ext_link}}
  11. If _st matches ^(404|410|5\d\d|unreachable)$
  12. broken_links = {{broken_links|0}} + 1
  13. Add row: link, status, found_on, kind
  14. If broken_links greater 0
  15. Notify: Broken links on {{start_url}}
  16. If alert_slack equals yes
  17. Slack: Broken links on {{start_url}} {{broken_links}} bro

Get this template

1. Download Develofly Autopilot: free, for Ubuntu and other Linux desktops.

Download Develofly Autopilot

2. The app comes with every template, so this one is already in your library. Click below to open it (or find it under Templates).

Open in Autopilot

What you need

  • A browser — the app installs it for you

You fill in: Start page, Most pages to crawl, Only pages under this path (optional), Also check links to other sites, Most external links to check, Also check images, Pause between pages (ms), Also send alerts to Slack.

Make it yours

Every step is editable in the app: change the site, the columns, the schedule, or chain it with other bots. Run it once, on a schedule, or sell it as your own bot.

More Website, SEO & Monitoring templates

SSL / domain expiry monitorWarn before certificates or domains expire.PageSpeed / Core Web Vitals loggerLog performance scores and Core Web Vitals for key URLs.SERP rank checkerTrack your keyword positions over time into a sheet.Sitemap diff watcherAlert when a site adds or removes pages from its sitemap.