Crawlab MCP

How to Install and Use Crawlab MCP

Crawlab MCP is a tool that helps AI applications talk to Crawlab for managing spiders, tasks, and files easily using natural language. Below is a simple guide to install and start using Crawlab MCP.

Installing Crawlab MCP as a Python Package

First, you can install Crawlab MCP quickly with Python.

To do this, you need to run one of the commands below. You can install it from the source code or from GitHub if available:

pip install -e .

Or, if the GitHub package is available, run:

pip install git+https://github.com/crawlab-team/crawlab-mcp-server.git

Once installed, you can use the command line interface (CLI) to start the server and client.

To start the server, run:

crawlab_mcp-mcp server [--spec PATH_TO_SPEC] [--host HOST] [--port PORT]

To start the client and connect to a running Crawlab MCP server, run:

crawlab_mcp-mcp client SERVER_URL

This method is the fastest for users comfortable with Python commands.

Running Crawlab MCP Locally with Python

If you prefer to run Crawlab MCP on your own computer without installing it as a package, you can do that too.

Make sure you have these ready before starting:

• Python version 3.8 or higher
• A running Crawlab instance you can connect to
• Your Crawlab API token

To configure the setup:

1. Copy the example environment file to a working file:

cp .env.example .env

1. Open the .env file with a text editor and update it with your Crawlab API URL and token like this:

CRAWLAB_API_BASE_URL=http://your-crawlab-instance:8080/api
CRAWLAB_API_TOKEN=your_api_token_here

To run locally:

1. Install needed packages by running:

pip install -r requirements.txt

1. Start the server by running:

python server.py

This will launch the Crawlab MCP server on your computer, ready to accept requests.

Running Crawlab MCP with Docker

If you prefer Docker, you can easily build and run Crawlab MCP in a container.

Follow these simple steps:

1. Build the Docker image with this command:

docker build -t crawlab-mcp-server .

1. Run the container, passing the environment variables from your .env file:

docker run -p 8000:8000 --env-file .env crawlab-mcp-server

This runs the MCP server and makes it available on port 8000 of your computer.

Adding Crawlab MCP to Docker Compose

If you already use Crawlab with Docker Compose, you can add the MCP server as a service.

Add this block to your docker-compose.yml file:

services:
  # ... existing Crawlab services

  mcp-server:
    build: ./backend/mcp-server
    ports:
      - "8000:8000"
    environment:
      - CRAWLAB_API_BASE_URL=http://backend:8000/api
      - CRAWLAB_API_TOKEN=your_api_token_here
    depends_on:
      - backend

This setup will build and run the MCP server together with your existing Crawlab services.

Using Crawlab MCP with AI Applications

Once your Crawlab MCP server is running, you can connect AI tools like Claude Desktop to it.

To connect:

1. Open Claude Desktop and go to Settings > MCP Servers.
2. Add a new server with the URL to your MCP server, for example http://localhost:8000.
3. Start chatting and use natural language commands to control Crawlab, such as:

• "Create a new spider named ‘Product Scraper’ for e-commerce"
• "Run the ‘Product Scraper’ spider on all nodes"
• "Show me the results of the last run"

Crawlab MCP turns these commands into API calls, manages spiders, tasks, and files for you.

---

By following these straightforward steps, you can install, configure, and start using Crawlab MCP easily. It helps AI apps interact naturally with Crawlab to perform common scraping and management tasks.