AnythingLLM MCP URL extraction guide
How to Fetch Web Pages in AnythingLLM with Webstractor MCP
AnythingLLM offers several ways to work with web content: built-in web tools, document ingestion, Agent Flows, and MCP. The MCP route is best for on-demand reading of one known public URL when the conversation needs current context but the page does not need to become a persistent workspace document.
Quick answer
Give AnythingLLM agents a reusable URL-to-context tool.
Register Webstractor with type streamable in anythingllm_mcp_servers.json, reload the server, and call extract_public_url from an agentic chat. Use it for exact public URLs when you want cleaned Markdown or normalized JSON without first importing the page as a permanent workspace document.
{
"mcpServers": {
"webstractor": {
"type": "streamable",
"url": "https://webstractor.com/mcp"
}
}
}Before you begin
What you need
- AnythingLLM with MCP support
- A workspace configured for agent tool use
- A public test URL
Step-by-step setup
Connect Webstractor to AnythingLLM
Choose on-demand extraction instead of ingestion
Use this workflow when a page is needed for the current task. If it must be searched repeatedly as part of a knowledge base, extract and review it first, then use AnythingLLM’s document workflow separately.
Locate the MCP server configuration
Open MCP Management or edit the plugins/anythingllm_mcp_servers.json file under AnythingLLM storage. In Docker, use the persistent storage mount.
Register the Streamable HTTP endpoint
AnythingLLM names this transport streamable. Add the server without removing existing entries.
{
"mcpServers": {
"webstractor": {
"type": "streamable",
"url": "https://webstractor.com/mcp"
}
}
}Reload and verify extract_public_url
Use the MCP Management controls to restart servers, inspect status, and confirm the extraction tool is listed.
Run an on-demand reading test
Prompt: “@agent Use Webstractor extract_public_url to read https://en.wikipedia.org/wiki/Retrieval-augmented_generation in Markdown. Explain the workflow and preserve the source URL.”
Architecture
On-demand context and workspace RAG solve different problems
MCP returns current page content to the active agent task. Workspace ingestion embeds a document for repeated retrieval later. Keeping those stages separate avoids filling a knowledge base with every page an exploratory agent touches.
A useful workflow is to fetch, inspect provenance and relevance, and only then choose whether to ingest.
Output strategy
Markdown for answers, JSON for flows
Ask for Markdown when the agent should summarize or quote the structure of a page. Request JSON when an Agent Flow needs stable fields, a semantic type, or collection items from supported sources.
- Known URL: extract_public_url
- Unknown source: search_web first
- Human-readable task: Markdown
- Programmatic branch: JSON
Troubleshooting
Inspect AnythingLLM’s MCP logs
The management UI exposes server state and logs. A JSON parse or transport error belongs to configuration; a single rejected URL usually belongs to public accessibility or content support.
If the agent uses its built-in scraper instead, explicitly name Webstractor during verification. Once behavior is confirmed, Intelligent Tool Selection can handle larger tool collections.
Available data
What you can extract
- On-demand Markdown from one public page
- Typed JSON for supported semantic entities
- Public metadata and canonical provenance
- Focused sections from long pages
AI workflows
Where normalized data helps
- Read a current policy page without ingesting it permanently
- Compare two public product pages in an agent session
- Summarize a release announcement with source provenance
- Selectively prepare pages for later RAG ingestion
Boundaries
Public data only
- MCP extraction is not the same as adding a document to the workspace vector database.
- Only public URLs are accepted; intranet and authenticated content are blocked.
- One tool call reads one URL rather than crawling linked pages.
webstractor.com does not bypass CAPTCHAs, login walls, paywalls, access controls, or regional restrictions. Review the source’s terms and applicable law before collecting or reusing data.
Common questions
AnythingLLM and Webstractor FAQ
Does fetched content automatically enter AnythingLLM RAG?
No. MCP tool output is conversation context. Adding a reviewed document to the workspace is a separate ingestion step.
Can this read a page behind my login?
No. Webstractor is limited to ordinary public URLs and does not receive browser sessions or credentials.
What should I use for a very long page?
Ask the agent to pass a narrow focus topic and request Markdown so less irrelevant content enters the model context.
Ready to try it?