What it does
Crawls your site from the start page (same site only, up to the page limit), finds every link that answers 404, 410, a server error or can't be reached, and lists each one with the page it's on. It can also check links to other sites and images. Results open in Excel; with alerts on you're told when something breaks.
How it works
- Opens the page
- Picks out the values
- Goes through each item
- Adds rows to your sheet
- Decides what to do next
- Calls the service's API
- Alerts you
- Sends a chat message
All 17 steps (as they appear in the app)
- Open {{start_url}}
- Run JavaScript → _crawl
- Read values: pages_checked, pages_not_checked, _broken, _external
- broken_links = 0
- For each bad in {{_broken}} (2 steps)
- broken_links = {{broken_links|0}} + 1
- Add row: link, status, found_on, kind
- If check_external equals yes
- For each ext in {{_external}} (2 steps)
- GET {{ext_link}}
- If _st matches ^(404|410|5\d\d|unreachable)$
- broken_links = {{broken_links|0}} + 1
- Add row: link, status, found_on, kind
- If broken_links greater 0
- Notify: Broken links on {{start_url}}
- If alert_slack equals yes
- Slack: Broken links on {{start_url}} {{broken_links}} bro
Get this template
1. Download Develofly Autopilot: free, for Ubuntu and other Linux desktops.
Download Develofly Autopilot2. The app comes with every template, so this one is already in your library. Click below to open it (or find it under Templates).
Open in AutopilotWhat you need
- A browser — the app installs it for you
You fill in: Start page, Most pages to crawl, Only pages under this path (optional), Also check links to other sites, Most external links to check, Also check images, Pause between pages (ms), Also send alerts to Slack.
Make it yours
Every step is editable in the app: change the site, the columns, the schedule, or chain it with other bots. Run it once, on a schedule, or sell it as your own bot.