webstractor

Qwen Code URL extraction / 2026

How to Fetch Web Pages in Qwen Code with Webstractor MCP in 2026

Web search discovers pages; URL extraction reads one. Keeping those jobs separate matters in a coding agent because it prevents broad searches when a user has already supplied the authoritative documentation URL. Qwen Code’s remote HTTP MCP support gives the agent a hosted page reader with no local scraper to install.

20264 min readFor agent builders

Quick answer

Connect once, then give Qwen Code the exact URL.

After adding the remote HTTP server, ask Qwen Code to call extract_public_url on a known public page. Request Markdown for analysis, add a short focus topic for long pages, and require the canonical source URL in its answer.

MCP configuration
qwen mcp add --transport http webstractor https://webstractor.com/mcp

Before you begin

What you need

  • Current Qwen Code CLI
  • A public URL to inspect
  • Tool-capable model configuration

Step-by-step setup

Connect Webstractor to Qwen Code

01

Add the server at the right scope

Use user scope for access across repositories or project scope for a repository-specific setup. The hosted endpoint uses Streamable HTTP.

Project-scope example
qwen mcp add --scope project --transport http webstractor https://webstractor.com/mcp
02

Verify tool discovery

Restart Qwen Code if it was open, enter /mcp, and inspect the webstractor server. Confirm that extract_public_url appears before giving the agent a larger task.

Open MCP manager
qwen
# Then enter /mcp
03

Run a known-URL test

Ask for Markdown and a narrow focus. Requiring quotes is less reliable than asking for a concise paraphrase with the URL, because the page may contain formatting or generated labels.

Validation prompt
# Paste into Qwen Code
Use webstractor extract_public_url to read https://qwenlm.github.io/qwen-code-docs/en/users/features/mcp/ in Markdown with focus "HTTP server configuration". Summarize the exact fields and cite the page URL.

Output choice

Markdown for reasoning, JSON for code paths

Markdown keeps headings, paragraphs, lists, and links in a form that fits Qwen Code’s context. Choose JSON if the next step is programmatic—such as extracting dates, semantic types, or feed items.

For multi-source tasks, fetch pages one at a time and label each source in the prompt. This makes disagreements and stale pages visible instead of blending them into one untraceable summary.

  • Use focus for one topic
  • Omit focus for a complete readable page
  • Always retain the returned source URL

Troubleshooting

When a URL returns little or no text

First try an ordinary public documentation page to separate client configuration from source availability. The original page may be login-only, removed, image-only, or hostile to automated readers.

If a focused result is too narrow, retry the same URL without focus. If /mcp does not list the tool at all, inspect the active settings scope and any `mcp.allowed` or `mcp.excluded` rules.

01

Available data

What you can extract

  • Clean Markdown from ordinary public pages
  • Focused sections from long pages
  • Schema-v1 JSON for structured processing
  • Source title, URL, author, and dates when available
02

AI workflows

Where normalized data helps

  • Read linked documentation before implementing a change
  • Summarize a public RFC or release post
  • Turn a reference page into implementation notes
  • Compare two known public pages with attribution

Boundaries

Public data only

  • One tool call reads one public URL and does not crawl links.
  • Focus is a relevance hint, not a completeness guarantee.
  • Private network URLs and authenticated pages are outside the tool.

webstractor.com does not bypass CAPTCHAs, login walls, paywalls, access controls, or regional restrictions. Review the source’s terms and applicable law before collecting or reusing data.

Common questions

Qwen Code and Webstractor FAQ

Does Qwen Code need to search before reading a URL?

No. When you already know the URL, ask it to call extract_public_url directly.

Does the tool execute actions on the website?

No. It reads public content and does not log in, submit forms, or automate site workflows.

Why use a focus topic?

Focus helps a long generic page prioritize a named topic, reducing unrelated context sent to the model.

Ready to try it?

Turn a public URL into useful context.

Enable page fetching Read the MCP guide