不想写代码? 使用 Web Access CLI 或 Scraping Skill,让您的智能体开箱即用地具备抓取能力。
将页面内容输入 LLM
抓取页面后,将 HTML 交给模型进行信息提取、摘要或分析,无需编写任何解析逻辑。Python + Claude
import requests
import anthropic
html = requests.get(
"https://parsing.webunlocker.gologin.com/v1/scrape",
params={"url": "https://news.ycombinator.com"},
headers={"apikey": "YOUR_API_KEY"}
).text
client = anthropic.Anthropic()
message = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=16000,
messages=[
{
"role": "user",
"content": f"Extract the top 5 story titles and URLs from this HTML:\n\n{html[:8000]}"
}
]
)
# the response can also contain thinking blocks, so print only the text
print("".join(block.text for block in message.content if block.type == "text"))
JavaScript + Claude
import Anthropic from "@anthropic-ai/sdk";
const html = await fetch(
"https://parsing.webunlocker.gologin.com/v1/scrape?url=" +
encodeURIComponent("https://news.ycombinator.com"),
{ headers: { apikey: "YOUR_API_KEY" } }
).then((r) => r.text());
const client = new Anthropic();
const message = await client.messages.create({
model: "claude-sonnet-5-5",
max_tokens: 16000,
messages: [
{
role: "user",
content: `Extract the top 5 story titles and URLs from this HTML:\n\n${html.slice(0, 8000)}`,
},
],
});
// the response can also contain thinking blocks, so print only the text
console.log(message.content.find((block) => block.type === "text")?.text);
为智能体提供抓取工具
定义一个由 Web Unlocker 支持的scrape 工具。智能体会自行决定何时获取页面,以及抓取哪个 URL。
import requests
import anthropic
def scrape(url: str) -> str:
"""Fetch a URL and return its rendered HTML."""
return requests.get(
"https://parsing.webunlocker.gologin.com/v1/scrape",
params={"url": url},
headers={"apikey": "YOUR_API_KEY"}
).text
tools = [
{
"name": "scrape",
"description": "Fetch the rendered HTML of any URL, including JavaScript-heavy and protected sites.",
"input_schema": {
"type": "object",
"properties": {
"url": {"type": "string", "description": "The URL to scrape"}
},
"required": ["url"]
}
}
]
client = anthropic.Anthropic()
messages = [
{"role": "user", "content": "What is the current price of the iPhone 16 Pro on apple.com?"}
]
while True:
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=16000,
tools=tools,
messages=messages
)
if response.stop_reason != "tool_use":
print("".join(block.text for block in response.content if block.type == "text"))
break
tool_results = []
for block in response.content:
if block.type == "tool_use" and block.name == "scrape":
html = scrape(block.input["url"])
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": html[:10000]
})
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": tool_results})
使用 Web Unlocker SDK
对于 TypeScript/Node.js 智能体,gologin-webunlocker SDK 提供了带类型的方法:
import { WebUnlocker } from "gologin-webunlocker";
const client = new WebUnlocker({
apiKey: process.env.GOLOGIN_WEBUNLOCKER_API_KEY!
});
// get structured metadata instead of raw HTML
const { data } = await client.scrapeJSON("https://example.com");
console.log(data.title, data.description, data.headings);
// get clean text for LLM context
const { text } = await client.scrapeText("https://example.com");
// batch scrape with concurrency control
const results = await client.batchScrape(
["https://example.com/1", "https://example.com/2", "https://example.com/3"],
{ concurrency: 3 }
);
使用 CLI
对于会调用 shell 命令的智能体(如 Claude Code),CLI 是最简单的方式:# plain text — good for LLM context
gologin-web-access scrape-text https://example.com
# structured JSON — title, description, headings, links
gologin-web-access scrape-json https://example.com
# batch scrape — many pages at once
gologin-web-access batch-scrape url1 url2 url3 --format text --summary
# search the web
gologin-web-access search "best antidetect browser 2026" --limit 10
# extract structured data with a schema
gologin-web-access extract https://example.com --schema ./schema.json
常见智能体模式
| 模式 | 实现方式 |
|---|---|
| 研究型智能体 | 智能体调用 scrape 工具 → 阅读页面 → 综合分析结果 |
| 价格监控 | 按计划对商品页面执行 batch-scrape → 与上一次运行结果对比 |
| 竞品情报 | 对竞争对手网站执行 crawl → 输入 LLM 进行分析 |
| 线索补全 | 对公司页面执行 scrape-json → 提取结构化数据 |
| 内容聚合 | search + scrape-text → 构建精选内容流 |
相关内容
- 什么是 Web Unlocker — 产品概览与定价
- Web Unlocker SDK — TypeScript SDK 参考
- Web Access CLI — 统一 CLI
- AI Skills — 适用于 Claude Code 的即插即用技能