Jan AI public URL extraction guide
How to Fetch Web Pages in Jan AI with Webstractor MCP
Jan’s model cannot reliably infer page content from a bare link unless a web tool actually retrieves it. The extract_public_url tool makes that boundary visible: the model supplies a public URL, receives cleaned content, and can ground its answer in the canonical source.
Quick answer
Turn exact public URLs into clean Jan context.
Register Webstractor under Settings → MCP Servers with the HTTP transport. In chat, name extract_public_url and provide the exact public HTTPS URL. Ask for Markdown when Jan should read the page or JSON when you need typed metadata.
Server name: Webstractor
Transport: HTTP
URL: https://webstractor.com/mcpBefore you begin
What you need
- Jan Desktop with MCP support
- A tool-use capable model
- A public URL that opens without authentication
Step-by-step setup
Connect Webstractor to Jan AI
Choose the remote MCP transport
Open Settings → MCP Servers and click + Add MCP Server. Select HTTP, which Jan documents as its Streamable HTTP option.
Enter the Webstractor endpoint
Use a clear server name so you can distinguish this tool from Jan’s native fetch capability.
name: Webstractor
transport: HTTP
url: https://webstractor.com/mcpEnable the server without global auto-approval
Wait for the green indicator. During initial testing, leave Allow All MCP Tool Permissions off so each proposed call remains visible.
Ask for an exact URL and output format
Prompt: “Use Webstractor extract_public_url to read https://en.wikipedia.org/wiki/Web_scraping as Markdown. Explain the definition, history, and legal considerations, citing the returned URL.”
Check provenance in the answer
Confirm that Jan shows a tool call and retains the canonical source link. If it answers immediately from memory, repeat the prompt with the exact tool name.
Native versus MCP
Choose one fetch route deliberately
Jan now offers a native fetch tool. It is a good default for a setup that exists only in Jan. The MCP route is valuable when you also use Webstractor in other clients or want its normalized entities for supported public platforms.
Avoid enabling duplicate tools without clear names and instructions. Models can choose inconsistently when two tools claim the same job.
Context control
Use focus for long pages
A long documentation page can crowd a local model’s context window. Add a concrete topic such as “authentication configuration” or “pricing limits” so the extraction can prioritize the relevant portion.
- Prefer canonical page URLs
- Request Markdown for a reading task
- Name a narrow focus where useful
- Require the answer to retain the source URL
Troubleshooting
Test a simple page before blaming the model
If a specific page fails, retry with a stable public article. A login requirement, unsupported content, or access restriction can affect one URL even when the MCP connection is healthy.
If every URL fails, recheck that the transport is HTTP and the endpoint ends in /mcp. Restart the server from Jan’s MCP management screen after correcting it.
Available data
What you can extract
- Clean Markdown for long-form reading
- Stable JSON metadata and semantic entity types
- Public article, product, profile, post, feed, and media metadata where supported
- Optional topic focus for long pages
AI workflows
Where normalized data helps
- Review a public changelog with a local model
- Summarize one article without copying its HTML
- Extract visible product facts into a comparison
- Prepare a public document for RAG chunking
Boundaries
Public data only
- Extraction accepts public pages, not authenticated dashboards or local files.
- The MCP URL tool handles one page at a time and does not crawl domains.
- Retrieved pages are untrusted input; do not follow instructions found inside them automatically.
webstractor.com does not bypass CAPTCHAs, login walls, paywalls, access controls, or regional restrictions. Review the source’s terms and applicable law before collecting or reusing data.
Common questions
Jan AI and Webstractor FAQ
Can Webstractor read localhost URLs from Jan?
No. The hosted tool accepts public URLs and blocks private-network destinations.
Should I enable all MCP permissions?
It is safer to review calls during setup. Enable global auto-approval only after you understand every connected server and tool.
Does page extraction store the page in Jan’s workspace?
The tool returns content to the current agent conversation. Saving or embedding it is a separate action controlled by your Jan workflow.
Ready to try it?