feat: 添加 Dashboard 完整日志监控系统 v1.1.0
✨ 新增功能 - 完整的日志记录系统（6 种日志级别） - 日志配置功能（可通过 config.json 控制） - 性能监控装饰器和请求日志中间件 - 7 个管理工具脚本 - 完整的文档和使用指南 🛠️ 管理工具 - start-with-log.sh: 启动脚本（带日志） - stop-dashboard.sh: 停止脚本 - view-logs.sh: 日志查看器 - monitor-logs.sh: 实时监控工具（支持多种过滤器） - analyze-logs.sh: 日志分析工具（自动生成报告） - demo-logging.sh: 功能演示脚本 - test-logging-config.sh: 配置测试工具 📊 日志特性 - 支持 INFO/SUCCESS/WARN/ERROR/DEBUG/PERF 6 种级别 - 自动记录启动过程、API 请求、性能统计 - 缓存命中情况追踪 - 分步性能监控 - 智能过滤器 ⚙️ 配置功能 - 可控制是否启用日志（默认：true） - 可设置日志级别（默认：INFO） - 可控制文件/控制台输出 - 支持动态配置（重启生效） 📚 文档 - LOGGING_GUIDE.md: 完整使用指南 - LOGGING_CONFIG.md: 配置说明文档 - LOGGING_CONFIG_QUICK.md: 快速配置指南 - 多个中文说明文档 🔒 安全 - 添加 .gitignore 排除敏感信息 - config.json（含 Token）不提交 - 日志文件不提交 - 示例配置使用占位符 ✅ 测试 - 语法检查通过 - 功能完整性验证 - 配置控制测试通过 - 文档完整性检查 详见 CHANGELOG_v1.1.0.md Made-with: Cursor
This commit is contained in:
fang
82
skills/web-search/SKILL.md
Normal file
82
skills/web-search/SKILL.md
Normal file
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: web-search
|
||||
description: 免费全网搜索技能。使用 DuckDuckGo 搜索引擎，无需 API Key。支持搜索 + 网页抓取 + 内容提取。
|
||||
metadata: { "openclaw": { "emoji": "🌐", "requires": { "bins": ["curl", "node"] } } }
|
||||
---
|
||||
|
||||
# Web Search — 免费全网搜索
|
||||
|
||||
无需任何 API Key 即可搜索互联网并提取网页内容。
|
||||
|
||||
## 1. 搜索（DuckDuckGo）
|
||||
|
||||
```bash
|
||||
node {baseDir}/scripts/search.mjs "搜索关键词"
|
||||
node {baseDir}/scripts/search.mjs "搜索关键词" -n 10
|
||||
node {baseDir}/scripts/search.mjs "搜索关键词" --region cn-zh
|
||||
```
|
||||
|
||||
### 搜索参数
|
||||
|
||||
| 参数 | 说明 | 默认值 |
|
||||
|------|------|--------|
|
||||
| 第一个参数 | 搜索关键词（必填） | - |
|
||||
| `-n <数量>` | 返回结果数量 | 10 |
|
||||
| `--region <区域>` | 搜索区域，如 `cn-zh`（中国）、`us-en`（美国） | `wt-wt`（全球） |
|
||||
|
||||
### 搜索示例
|
||||
|
||||
```bash
|
||||
# 基础搜索
|
||||
node {baseDir}/scripts/search.mjs "加密货币钱包泄露"
|
||||
|
||||
# 限定区域和数量
|
||||
node {baseDir}/scripts/search.mjs "crypto wallet leak 2026" -n 15 --region us-en
|
||||
|
||||
# 搜索特定网站内容
|
||||
node {baseDir}/scripts/search.mjs "site:github.com wallet private key exposed"
|
||||
```
|
||||
|
||||
## 2. 网页内容抓取
|
||||
|
||||
```bash
|
||||
node {baseDir}/scripts/fetch.mjs "https://example.com/article"
|
||||
node {baseDir}/scripts/fetch.mjs "https://example.com" --raw
|
||||
```
|
||||
|
||||
### 抓取参数
|
||||
|
||||
| 参数 | 说明 | 默认值 |
|
||||
|------|------|--------|
|
||||
| 第一个参数 | 目标 URL（必填） | - |
|
||||
| `--raw` | 输出原始 HTML 而非提取的文本 | false |
|
||||
| `--max <字数>` | 最大输出字数 | 5000 |
|
||||
|
||||
## 典型工作流
|
||||
|
||||
对于需要全网调查的任务，按以下步骤执行：
|
||||
|
||||
1. **搜索**：用 `search.mjs` 搜索相关关键词，获取 URL 列表
|
||||
2. **抓取**：用 `fetch.mjs` 逐个读取感兴趣的页面内容
|
||||
3. **分析**：综合所有信息，给出结论
|
||||
4. **多轮搜索**：如果第一轮结果不够，换关键词再搜
|
||||
|
||||
### 完整示例：调查加密钱包泄露
|
||||
|
||||
```bash
|
||||
# 第一步：搜索
|
||||
node {baseDir}/scripts/search.mjs "crypto wallet private key leaked 2026"
|
||||
|
||||
# 第二步：针对找到的 URL 抓取详情
|
||||
node {baseDir}/scripts/fetch.mjs "https://example.com/security-report"
|
||||
|
||||
# 第三步：换个关键词继续搜
|
||||
node {baseDir}/scripts/search.mjs "数字钱包 私钥泄露 安全事件"
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 此技能完全免费，无需配置任何 API Key
|
||||
- DuckDuckGo 搜索可能有速率限制，如遇到限制请稍等片刻重试
|
||||
- 网页抓取不执行 JavaScript，部分动态页面可能无法获取完整内容
|
||||
- 建议使用英文关键词获得更全面的搜索结果
|
||||
5
skills/web-search/_meta.json
Normal file
5
skills/web-search/_meta.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"slug": "web-search",
|
||||
"version": "1.0.0",
|
||||
"publishedAt": 1741190400000
|
||||
}
|
||||
150
skills/web-search/scripts/fetch.mjs
Normal file
150
skills/web-search/scripts/fetch.mjs
Normal file
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* 网页内容抓取 — 将网页 HTML 转换为可读文本
|
||||
* 支持提取正文、标题、链接等关键信息
|
||||
*/
|
||||
|
||||
function usage() {
|
||||
console.error('Usage: fetch.mjs "URL" [--raw] [--max 5000]');
|
||||
console.error(" --raw Output raw HTML instead of extracted text");
|
||||
console.error(" --max <chars> Maximum output characters (default: 5000)");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
if (args.length === 0 || args[0] === "-h" || args[0] === "--help") usage();
|
||||
|
||||
const url = args[0];
|
||||
let raw = false;
|
||||
let maxChars = 5000;
|
||||
|
||||
for (let i = 1; i < args.length; i++) {
|
||||
if (args[i] === "--raw") raw = true;
|
||||
else if (args[i] === "--max") maxChars = parseInt(args[++i] || "5000", 10);
|
||||
}
|
||||
|
||||
function htmlToText(html) {
|
||||
let text = html;
|
||||
|
||||
text = text.replace(/<script[\s\S]*?<\/script>/gi, "");
|
||||
text = text.replace(/<style[\s\S]*?<\/style>/gi, "");
|
||||
text = text.replace(/<nav[\s\S]*?<\/nav>/gi, "");
|
||||
text = text.replace(/<footer[\s\S]*?<\/footer>/gi, "");
|
||||
text = text.replace(/<header[\s\S]*?<\/header>/gi, "");
|
||||
text = text.replace(/<!--[\s\S]*?-->/g, "");
|
||||
|
||||
const titleMatch = html.match(/<title[^>]*>([\s\S]*?)<\/title>/i);
|
||||
const title = titleMatch
|
||||
? titleMatch[1].replace(/<[^>]+>/g, "").trim()
|
||||
: "";
|
||||
|
||||
const metaDescMatch = html.match(
|
||||
/<meta[^>]*name=["']description["'][^>]*content=["']([\s\S]*?)["']/i,
|
||||
);
|
||||
const metaDesc = metaDescMatch ? metaDescMatch[1].trim() : "";
|
||||
|
||||
text = text.replace(/<h([1-6])[^>]*>([\s\S]*?)<\/h\1>/gi, (_, level, content) => {
|
||||
const prefix = "#".repeat(parseInt(level));
|
||||
return `\n${prefix} ${content.replace(/<[^>]+>/g, "").trim()}\n`;
|
||||
});
|
||||
|
||||
text = text.replace(/<p[^>]*>([\s\S]*?)<\/p>/gi, (_, content) => {
|
||||
return `\n${content.replace(/<[^>]+>/g, "").trim()}\n`;
|
||||
});
|
||||
|
||||
text = text.replace(/<li[^>]*>([\s\S]*?)<\/li>/gi, (_, content) => {
|
||||
return `- ${content.replace(/<[^>]+>/g, "").trim()}\n`;
|
||||
});
|
||||
|
||||
text = text.replace(
|
||||
/<a[^>]*href=["']([^"']+)["'][^>]*>([\s\S]*?)<\/a>/gi,
|
||||
(_, href, content) => {
|
||||
const linkText = content.replace(/<[^>]+>/g, "").trim();
|
||||
if (!linkText || href.startsWith("#") || href.startsWith("javascript:"))
|
||||
return linkText;
|
||||
return `[${linkText}](${href})`;
|
||||
},
|
||||
);
|
||||
|
||||
text = text.replace(/<br\s*\/?>/gi, "\n");
|
||||
text = text.replace(/<[^>]+>/g, "");
|
||||
|
||||
text = text.replace(/&/g, "&");
|
||||
text = text.replace(/</g, "<");
|
||||
text = text.replace(/>/g, ">");
|
||||
text = text.replace(/"/g, '"');
|
||||
text = text.replace(/'/g, "'");
|
||||
text = text.replace(/ /g, " ");
|
||||
|
||||
text = text.replace(/\n{3,}/g, "\n\n");
|
||||
text = text.replace(/[ \t]+/g, " ");
|
||||
text = text
|
||||
.split("\n")
|
||||
.map((line) => line.trim())
|
||||
.join("\n");
|
||||
|
||||
let result = "";
|
||||
if (title) result += `# ${title}\n\n`;
|
||||
if (metaDesc) result += `> ${metaDesc}\n\n`;
|
||||
result += text.trim();
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
try {
|
||||
const resp = await fetch(url, {
|
||||
headers: {
|
||||
"User-Agent":
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
Accept:
|
||||
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||
},
|
||||
redirect: "follow",
|
||||
signal: AbortSignal.timeout(15000),
|
||||
});
|
||||
|
||||
if (!resp.ok) {
|
||||
console.error(`HTTP ${resp.status}: ${resp.statusText}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const contentType = resp.headers.get("content-type") || "";
|
||||
if (
|
||||
!contentType.includes("text/html") &&
|
||||
!contentType.includes("text/plain") &&
|
||||
!contentType.includes("application/json")
|
||||
) {
|
||||
console.error(`不支持的内容类型: ${contentType}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const html = await resp.text();
|
||||
|
||||
if (raw) {
|
||||
console.log(html.slice(0, maxChars));
|
||||
} else if (contentType.includes("application/json")) {
|
||||
try {
|
||||
const json = JSON.parse(html);
|
||||
console.log(JSON.stringify(json, null, 2).slice(0, maxChars));
|
||||
} catch {
|
||||
console.log(html.slice(0, maxChars));
|
||||
}
|
||||
} else {
|
||||
const text = htmlToText(html);
|
||||
console.log(text.slice(0, maxChars));
|
||||
if (text.length > maxChars) {
|
||||
console.log(`\n...(内容已截断，共 ${text.length} 字符，显示前 ${maxChars} 字符)`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(`\n---\nURL: ${url}`);
|
||||
} catch (err) {
|
||||
if (err.name === "TimeoutError") {
|
||||
console.error("请求超时（15秒）");
|
||||
} else {
|
||||
console.error(`抓取失败: ${err.message}`);
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
149
skills/web-search/scripts/search.mjs
Normal file
149
skills/web-search/scripts/search.mjs
Normal file
@@ -0,0 +1,149 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* DuckDuckGo HTML 搜索 — 免费，无需 API Key
|
||||
* 通过解析 DuckDuckGo HTML Lite 页面获取搜索结果
|
||||
*/
|
||||
|
||||
function usage() {
|
||||
console.error('Usage: search.mjs "query" [-n 10] [--region wt-wt]');
|
||||
console.error(" -n <count> Number of results (default: 10)");
|
||||
console.error(" --region <r> Region code: cn-zh, us-en, wt-wt (default)");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
if (args.length === 0 || args[0] === "-h" || args[0] === "--help") usage();
|
||||
|
||||
const query = args[0];
|
||||
let maxResults = 10;
|
||||
let region = "wt-wt";
|
||||
|
||||
for (let i = 1; i < args.length; i++) {
|
||||
if (args[i] === "-n") {
|
||||
maxResults = parseInt(args[++i] || "10", 10);
|
||||
} else if (args[i] === "--region") {
|
||||
region = args[++i] || "wt-wt";
|
||||
}
|
||||
}
|
||||
|
||||
async function searchDDG(query, maxResults, region) {
|
||||
const tokenUrl = "https://duckduckgo.com/";
|
||||
const tokenResp = await fetch(tokenUrl, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/x-www-form-urlencoded",
|
||||
"User-Agent":
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
|
||||
},
|
||||
body: `q=${encodeURIComponent(query)}`,
|
||||
});
|
||||
const tokenHtml = await tokenResp.text();
|
||||
|
||||
const vqdMatch = tokenHtml.match(/vqd=["']([^"']+)["']/);
|
||||
if (!vqdMatch) {
|
||||
const vqdAlt = tokenHtml.match(/vqd=([\d-]+)/);
|
||||
if (!vqdAlt) {
|
||||
throw new Error("Failed to get DuckDuckGo search token");
|
||||
}
|
||||
var vqd = vqdAlt[1];
|
||||
} else {
|
||||
var vqd = vqdMatch[1];
|
||||
}
|
||||
|
||||
const searchUrl =
|
||||
`https://links.duckduckgo.com/d.js?` +
|
||||
`q=${encodeURIComponent(query)}` +
|
||||
`&kl=${region}` +
|
||||
`&vqd=${vqd}` +
|
||||
`&o=json&dl=en&ct=US&sp=0&bpa=1&biaexp=b&msvrtexp=b`;
|
||||
|
||||
const searchResp = await fetch(searchUrl, {
|
||||
headers: {
|
||||
"User-Agent":
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
|
||||
Referer: "https://duckduckgo.com/",
|
||||
},
|
||||
});
|
||||
|
||||
const text = await searchResp.text();
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(text);
|
||||
} catch {
|
||||
return fallbackHTMLSearch(query, maxResults);
|
||||
}
|
||||
|
||||
const results = (data.results || [])
|
||||
.filter((r) => r.u && r.t && !r.u.startsWith("https://duckduckgo.com"))
|
||||
.slice(0, maxResults)
|
||||
.map((r) => ({
|
||||
title: r.t.replace(/<\/?b>/g, ""),
|
||||
url: r.u,
|
||||
snippet: (r.a || "").replace(/<\/?b>/g, ""),
|
||||
}));
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
async function fallbackHTMLSearch(query, maxResults) {
|
||||
const url = `https://html.duckduckgo.com/html/?q=${encodeURIComponent(query)}`;
|
||||
const resp = await fetch(url, {
|
||||
headers: {
|
||||
"User-Agent":
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
|
||||
},
|
||||
});
|
||||
const html = await resp.text();
|
||||
|
||||
const results = [];
|
||||
const linkRegex =
|
||||
/<a[^>]*class="result__a"[^>]*href="([^"]*)"[^>]*>([\s\S]*?)<\/a>/g;
|
||||
const snippetRegex =
|
||||
/<a[^>]*class="result__snippet"[^>]*>([\s\S]*?)<\/a>/g;
|
||||
|
||||
const links = [...html.matchAll(linkRegex)];
|
||||
const snippets = [...html.matchAll(snippetRegex)];
|
||||
|
||||
for (let i = 0; i < Math.min(links.length, maxResults); i++) {
|
||||
let rawUrl = links[i][1];
|
||||
try {
|
||||
const urlObj = new URL(rawUrl, "https://duckduckgo.com");
|
||||
const uddg = urlObj.searchParams.get("uddg");
|
||||
if (uddg) rawUrl = decodeURIComponent(uddg);
|
||||
} catch {}
|
||||
|
||||
results.push({
|
||||
title: links[i][2].replace(/<[^>]+>/g, "").trim(),
|
||||
url: rawUrl,
|
||||
snippet: (snippets[i]?.[1] || "").replace(/<[^>]+>/g, "").trim(),
|
||||
});
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
try {
|
||||
const results = await searchDDG(query, maxResults, region);
|
||||
|
||||
if (results.length === 0) {
|
||||
console.log("未找到搜索结果。请尝试不同的关键词。");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log(`## 搜索结果：「${query}」\n`);
|
||||
console.log(`共 ${results.length} 条结果\n`);
|
||||
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i];
|
||||
console.log(`### ${i + 1}. ${r.title}`);
|
||||
console.log(` URL: ${r.url}`);
|
||||
if (r.snippet) {
|
||||
console.log(` ${r.snippet}`);
|
||||
}
|
||||
console.log();
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`搜索失败: ${err.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
Reference in New Issue
Block a user