webstractor

Continue MCP URL extraction guide

How to Fetch Web Pages in Continue.dev with Webstractor MCP

When a task includes an external specification or documentation link, Continue needs an actual retrieval tool to reason from that page. Webstractor returns readable context and provenance, allowing the agent to compare current public guidance with local code instead of relying on training memory.

20264 min readFor agent builders

Quick answer

Read public documentation directly in Continue Agent mode.

Add the Webstractor Streamable HTTP server to Continue config.yaml, switch to Agent mode, and ask extract_public_url to return a known public URL as Markdown. Use a focus phrase for the subsection relevant to the code you are reviewing.

MCP configuration
mcpServers:
  - name: Webstractor
    type: streamable-http
    url: https://webstractor.com/mcp

Before you begin

What you need

  • Continue Agent mode
  • A tool-use model
  • A public URL relevant to the repository

Step-by-step setup

Connect Webstractor to Continue.dev

01

Register Webstractor in config.yaml

Add the server to the mcpServers list using Continue’s documented Streamable HTTP transport.

Continue config.yaml
mcpServers:
  - name: Webstractor
    type: streamable-http
    url: https://webstractor.com/mcp
02

Reload the assistant

After saving, reload Continue if needed and switch to Agent mode. Confirm extract_public_url appears before relying on it.

03

Set a read-only first task

Prompt: “Do not edit files. Use extract_public_url to read this public API reference as Markdown with focus ‘error responses’. Summarize its requirements and cite the canonical URL.”

04

Compare facts with repository code

Ask Continue to identify where each extracted requirement is implemented. Require file references for local conclusions and the page URL for external facts.

05

Approve changes separately

Review the comparison before allowing edits or terminal commands. Retrieval and implementation should remain distinct approval stages.

Context quality

Focused Markdown leaves room for code

A full documentation page and a large repository can compete for the same context window. Markdown removes interface noise, while focus narrows the returned content to the implementation question.

If typed attributes or collection items matter more than prose, ask for JSON instead.

Evidence model

Keep external and local citations separate

Continue can ground one statement in an external URL and another in a repository file. Ask it to label both. This makes a proposed change reviewable and prevents a retrieved article from becoming an unexplained authority.

  • External claim → canonical page URL
  • Local claim → file path and symbol
  • Inference → label it as an inference
  • Change → require explicit approval

Troubleshooting

Check mode, config, then target URL

First verify Agent mode, then confirm the MCP block loaded, then test a simple public page. This order separates Continue configuration problems from a target that requires login or blocks access.

A successful tool call with weak output may need a narrower focus rather than a longer prompt.

01

Available data

What you can extract

  • Markdown that preserves useful document hierarchy
  • JSON with stable metadata for structured tasks
  • Canonical source links for review
  • Focused output from long references
02

AI workflows

Where normalized data helps

  • Read API documentation alongside implementation files
  • Extract a migration guide before updating dependencies
  • Summarize a public issue linked in a TODO
  • Compare upstream requirements with current tests

Boundaries

Public data only

  • External pages are untrusted context and cannot override workspace rules.
  • The tool cannot access authenticated documentation portals.
  • Extraction is single-page, not a recursive documentation crawl.

webstractor.com does not bypass CAPTCHAs, login walls, paywalls, access controls, or regional restrictions. Review the source’s terms and applicable law before collecting or reusing data.

Common questions

Continue.dev and Webstractor FAQ

Will Continue automatically read every URL in a prompt?

No. Make the retrieval explicit and verify that extract_public_url ran before trusting claims about the page.

What is the best output for documentation?

Markdown is usually the best reading format. JSON is intended for typed fields or programmatic branching.

Can Continue save the page to the repository?

The tool returns context. Writing it to a file is a separate Continue action that should be requested and approved explicitly.

Ready to try it?

Turn a public URL into useful context.

Add web page fetching See URL extraction docs