AnyCrawl MCP

How to Install and Use AnyCrawl MCP

If you want to use AnyCrawl MCP, a powerful web scraping and crawling tool, here is a simple and clear guide to get started. We will cover how to install it and run some basic examples. Follow these easy steps to have AnyCrawl MCP up and running quickly.

Setting Up Your Environment

Before installing AnyCrawl MCP, you need to get an API key. This key lets you access the service.

1. Visit the AnyCrawl website: https://anycrawl.dev.
2. Sign up for a free account.
3. After logging in, open your dashboard.
4. Find the API Keys section.
5. Copy your API key.
6. On your computer, set this key as an environment variable using:

export ANYCRAWL_API_KEY="your-api-key-here"

This step is important because it allows AnyCrawl MCP to authenticate your requests.

Installing AnyCrawl MCP

You have two main options to install and run AnyCrawl MCP: using npx for quick runs or installing it manually.

Running with npx (Quick Setup)

If you want to start using AnyCrawl MCP right away without installing it globally, run:

ANYCRAWL_API_KEY=your-api-key-here npx -y anycrawl-mcp

This command downloads and runs AnyCrawl MCP directly without permanent installation.

Manual Installation

To install AnyCrawl MCP globally on your system, use npm:

npm install -g anycrawl-mcp-server

After installation, you can start the server using:

ANYCRAWL_API_KEY=your-api-key-here anycrawl-mcp

Running AnyCrawl MCP

AnyCrawl MCP supports different running modes depending on what you need.

• The default mode uses standard input/output (STDIO).
• You can also run it in MCP mode for HTTP communication.
• Or use SSE mode for server-sent events (great for web apps).

Here are some useful commands to start AnyCrawl MCP in different modes for development:

npm run dev        # STDIO mode (default)
npm run dev:mcp    # MCP HTTP mode
npm run dev:sse    # SSE mode for web apps

To run in production, use:

npm start          # STDIO mode
npm run start:mcp  # MCP HTTP mode
npm run start:sse  # SSE mode

You can also set the mode by setting the environment variable, for example:

ANYCRAWL_MODE=MCP ANYCRAWL_API_KEY=your-api-key npm run dev:mcp

Example: Scrape a Single Web Page

Once AnyCrawl MCP is running, you can use tools like anycrawl_scrape to scrape content from a single URL. Here is an example command payload to scrape a page using the Cheer.io engine and get markdown and HTML formats:

{
  "name": "anycrawl_scrape",
  "arguments": {
    "url": "https://example.com",
    "engine": "cheerio",
    "formats": ["markdown", "html"],
    "timeout": 30000
  }
}

This example tells the server to get the content from https://example.com quickly and return it as markdown and HTML.

Wrapping Up

With these steps, you can install and start using AnyCrawl MCP for web scraping and crawling. Make sure you always set your API key properly in the environment, then choose the installation and running mode that fits your needs. The tool offers flexible commands and modes to help you build powerful web data extraction solutions.