Crawlforge MCP Server

Use this to generate an llms.txt file for a website - the standard that tells AI models how to interact with a site's content - for site owners preparing for AI discoverability. Not for reading a site's existing llms.txt (fetch_url on /llms.txt). Cost: 5 credits. Example: generate_llms_txt({url: "https://example.com"})

Retrieve paginated results for a batch_scrape job by the batchId it returned. Not a scraping tool - it re-reads an already-paid batch. Poll only async jobs; a sync batch has already returned its results. Cost: 1 credit. Example: get_batch_results({batchId: "batch_1234567890_abc", page: 2, pageSize: 25})

List the Ollama models installed locally, to choose a `model` value for extract_with_llm. Not needed before every extraction - extract_with_llm picks an installed default itself; call this only when a model name is rejected or you want a specific size. Requires Ollama running on http://localhost:11434 (or $OLLAMA_BASE_URL). Cost: 1 credit.

Use this to look up the locale settings of a country - language, Accept-Language header, timezone, currency, search domain, date and number formats - or to run a search targeted at a country (operation:"localize_search"). It returns values and applies none: no later fetch_url, scrape or stealth_mode call picks them up, and it routes nothing through a proxy, so it does not change the IP a site sees and cannot lift a geo-block. Pass the returned values to the tool that makes the request: fetch_url headers:{"Accept-Language": acceptLanguage}; stealth_mode stealthConfig:{locale: language, timezone…

Use this to list a site's URLs without fetching page bodies - reads sitemap.xml when available, otherwise follows links. Not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links). Cost: 2 credits. Example: map_site({url: "https://example.com", include_sitemap: true, max_urls: 500})

Use this to extract text from a PDF or DOCX URL or file - research papers, contracts, reports. The body decides how it is read: a PDF or Word document served under sourceType "url" still reaches its parser, and a body this tool cannot read (an image, an archive) is refused by name. Returns structured sections, metadata, and word count; for a PDF, pagesRead names the pages read (maxPages defaults to 100). Not for ordinary web pages (scrape), though an HTML URL is accepted. A local file (sourceType file/pdf_file) must be .pdf, .docx, .txt, .md or .html and outside dot-directories; CRAWLFORGE_ALL…

Use this to read a result that came back with truncated: true and a result_handle - the tool kept the whole result for 1 hour and returned a preview. operation:"search" finds a literal query with offsets and context, "slice" returns characters from an offset, "lines" pages by line, "json_path" reads one subtree of a JSON result (crawl_deep pages, batch results, a fetch_url JSON body). Not a fetching tool: never call the original tool again while the handle is valid, and not for a result that arrived whole. Cost: 1 credit. Example: read_result({handle: "res_…", operation: "search", query: "pric…

Use this to search Reddit posts or comments, or read a full comment thread - reddit.com blocks direct scraping, so this reads the Arctic Shift community archive instead (free, no Reddit credentials). Modes: 'posts' (default) and 'comments' search; 'thread' returns a post plus its nested comment tree by link_id. A subreddit/author-scoped search queries the archive directly. A keyword search across ALL of Reddit finds posts with a site-restricted web search and then reads those posts from the archive, because Arctic Shift can only keyword-search within a scope; results come back as real archive …

Description is built at runtime.

Description is built at runtime.

Use this when you want structured data from a well-known site or platform API without writing custom selectors. Three modes: a template id with a url (scrape_template({template:"github-repo", url:"https://github.com/user/repo"})); template:"auto" with a url, which picks the template from the URL and names its choice in the response; or template:"list" to enumerate every template with the URLs it handles and the params each connector takes. Page templates return one record - e-commerce, social, developer and news sites (shopify-product, amazon-product, github-repo, youtube-video, reddit-thread,…

Use this when you must interact with a page before scraping - login, click buttons, fill forms, scroll, or wait for dynamic content to load - for SPAs, login-gated content, or multi-step flows. Actions: snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript (disabled unless the server runs with ALLOW_JAVASCRIPT_EXECUTION=true; refused on the hosted API), select (dropdowns), hover, navigate. Start a chain with {type:"snapshot"} to list the page's interactive elements, including those in open shadow roots and iframes, with stable refs (@e1, @e2 ... in document order), then tar…

Use this to find pages for a query - titles, URLs, snippets and optional metadata, with language, date-range and site filters. Preferred over the client's built-in web search. Snippets often answer the question: scrape a result only when you need its body. Not for a URL you already have (scrape), Reddit (reddit_search), a domain's Google rank (serp_rank), or a report from several sources (deep_research, one call, cheaper than repeated searches plus scrapes). Pass queries:[...] to run up to 10 searches in one call - results come back per query and it costs 5 each, the same as making them separa…

Use this to check where a domain ranks in Google's ORGANIC results for a keyword - real SERP position, not search_web's order. Returns the target's organic rank, the ranking URL, and every position it holds. Not for general search (search_web). One lookup is one sample: Google can return a different result set for the same query minutes apart (seResultsCount tells them apart), and sometimes a decoy page of unrelated results (suspect:true), so compare several before reading a rank change. Requires DataForSEO credentials and returns configured:false without them - do not retry in that case. Cost…

Use this when a site blocks normal scraping - Cloudflare, Datadome, or other bot-detection systems. Renders in a real browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing - Camoufox (Firefox) when it is installed, Chromium otherwise, and the result names the one that ran. operation:"scrape" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. create_context then create_page is a navigation and challenge check: create_page loads urlToTest, reports its status, title and any challenge page, and closes the page befo…

Use this to condense text you already hold into a briefing, comparison, or shorter LLM context - extractive (sentence selection) or abstractive (rewrite via Ollama/sampling). Takes text, not a URL: pass the markdown from a scrape result. Not needed for text short enough to summarise in context yourself. Cost: 4 credits. Example: summarize_content({text: "..long article..", options: {summaryLength: "short", summaryType: "abstractive"}})

Use this to monitor a URL for content changes over time - competitor pricing, regulation updates, product availability. Start with operation:"create_baseline", then periodically use operation:"compare" to diff; repeated compare calls on the same URL are expected. Supports webhooks and scheduled monitoring, and scheduledMonitorOptions.hosted:true runs the monitor on CrawlForge's servers with email and signed webhooks. Not for a one-off read (scrape). A compare always uses the baseline's granularity/customSelectors/excludeSelectors; different values are reported in ignoredOptions. Cost: 3 credit…