Crawler

What this skill does

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

Crawler is part of the Developer Tools category — developer tools that help your agent write, review, debug, and ship code. You can install it on its own or alongside other developer tools skills from the OpenClaw catalog.

How to install Crawler

The easiest path is via the OpenClaw Easy desktop app — one click, no terminal required:

1. Download OpenClaw Easy for macOS or Windows (free, one-click installer, ~30 seconds).
2. Open the in-app Skills panel.
3. Search for crawler and click Install.
4. The skill activates automatically when an incoming message matches its description.

Install from the command line

If you already run the OpenClaw CLI, add Crawler with a single command:

openclaw skills add crawler

This pulls crawler from ClawHub and installs it into ~/.openclaw/skills/crawler/. Restart the OpenClaw gateway afterwards so the new skill is discovered.

How to use Crawler

Once installed, Crawler activates on its own: when an incoming message on WhatsApp, Telegram, Slack, Discord, Feishu or Line matches the skill's description, your OpenClaw agent loads it and runs the workflow. You can also trigger it explicitly by describing the task in chat. No extra configuration is required after install.

Manual install (advanced)

If you prefer manual installation:

1. Click the Download .zip button above to grab crawler-3.0.0.zip directly from our S3 mirror.
2. Unzip into ~/.openclaw/skills/crawler/ (create the directory if it does not exist).
3. Restart OpenClaw Easy (or the OpenClaw CLI gateway) so the new skill is discovered.

Frequently asked questions

How do I install Crawler?

Install Crawler in the OpenClaw Easy desktop app by opening the Skills panel, searching for crawler, and clicking Install. From a terminal you can run: openclaw skills add crawler. Either way the skill is placed in ~/.openclaw/skills/crawler/.

Is Crawler free?

Yes. Crawler is free and open-source, distributed under the Apache-2.0 license through ClawHub. No account or payment is required to download or run it.

What does Crawler do?

Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations

Related: more developer tools skills

If Crawler looks useful, you may also want to check out other developer tools skills in the OpenClaw catalog:

• Prospect Researcher ⬇ 2K
• Literature Manager ⬇ 2K
• Credex Protocol ⬇ 2K
• MegaETH AI Developer ⬇ 2K
• Agent Skills ⬇ 2K

Browse the full OpenClaw skill catalog

This page covers just one skill. The OpenClaw skill hub has 10,000+ more — search, sort by downloads or stars, and install any of them in one click. There is also a curated awesome-openclaw-skills list grouped by use case.