更新: 21 个文件 - 2026-03-17 00:00:00
这个提交包含在:
@@ -164,7 +164,7 @@ func (s *SQLiExploit) TestErrorBased(payloads []struct {
|
||||
continue
|
||||
}
|
||||
|
||||
for dbms, pattern := range errorPatterns {
|
||||
for dbms := range errorPatterns {
|
||||
if strings.Contains(body, "SQL") || strings.Contains(body, "error") ||
|
||||
strings.Contains(body, "Error") || strings.Contains(body, "Warning") {
|
||||
results = append(results, InjectionResult{
|
||||
|
||||
@@ -14,9 +14,12 @@
|
||||
## 当前内容
|
||||
|
||||
- 工具: [xss-fuzzer.py](/Users/x/websafe/02-xss/tools/xss-fuzzer.py), [xss-scanner.go](/Users/x/websafe/02-xss/tools/xss-scanner.go)
|
||||
- 主题扩展: [前端与框架案例](/Users/x/websafe/07-framework-security/frontend-js/README.md)
|
||||
- 主题扩展: [frameworks/README.md](/Users/x/websafe/07-framework-security/frameworks/README.md)
|
||||
- 重点系统: [React](/Users/x/websafe/07-framework-security/frameworks/react/README.md), [Next.js](/Users/x/websafe/07-framework-security/frameworks/nextjs/README.md), [Vue](/Users/x/websafe/07-framework-security/frameworks/vue/README.md), [Nuxt](/Users/x/websafe/07-framework-security/frameworks/nuxt/README.md), [Vite](/Users/x/websafe/07-framework-security/frameworks/vite/README.md)
|
||||
- 实证链路: `xss-fuzzer/xss-scanner -> Playwright 回放 -> run bundle -> case/index 回写`
|
||||
|
||||
## 当前缺口
|
||||
## 当前状态
|
||||
|
||||
- `defense/`, `exploitation/`, `payloads/` 仍需补充实验专用内容
|
||||
- CSP、Trusted Types、Token 存储和前端敏感配置暴露已经转入 [07-framework-security/frontend-js](/Users/x/websafe/07-framework-security/frontend-js/README.md)
|
||||
- `defense/`, `exploitation/`, `payloads/` 仍保留实验载荷与说明位,但主题主索引已迁到 `07-framework-security/frameworks/*`
|
||||
- CSP、Trusted Types、Token 存储和前端敏感配置暴露通过系统页和 `05-defense/secure-code/*` 反向关联
|
||||
- 前端类 case 默认要求浏览器层证据;只有 HTTP 命中而无回放时,不记为 `verified-*`
|
||||
|
||||
@@ -15,8 +15,9 @@
|
||||
|
||||
- 暴力破解工具: [web-brute.py](/Users/x/websafe/03-authentication/bruteforce/tools/web-brute.py)
|
||||
- JWT 工具: [jwt-cracker.py](/Users/x/websafe/03-authentication/jwt/tools/jwt-cracker.py)
|
||||
- 会话边界工具: [session-lab.py](/Users/x/websafe/03-authentication/session/tools/session-lab.py)
|
||||
- 会话与 Token 风险案例: [福建案例总结](/Users/x/websafe/06-case-studies/fujian-gov-procurement/lessons-learned.md)
|
||||
|
||||
## 说明
|
||||
|
||||
该目录聚焦“验证控制面是否存在”而不是“最大化拿下账户”。对公网授权目标的验证应优先采用小样本、低频和可审计的实验方法。
|
||||
该目录聚焦“验证控制面是否存在”而不是“最大化拿下账户”。对公网授权目标的验证应优先采用小样本、低频和可审计的实验方法,并把浏览器存储、Cookie 属性和会话固定证据回写到 run bundle。
|
||||
|
||||
@@ -1,5 +1,18 @@
|
||||
# 会话实验
|
||||
|
||||
> `LAB NOTE` | `规划中`
|
||||
> `LAB ONLY` | `AUTHORIZED TARGETS ONLY`
|
||||
|
||||
该目录预留给 Cookie 属性、会话固定、登出失效、Token 轮换与浏览器存储对照实验。当前由 [07-framework-security/frontend-js](/Users/x/websafe/07-framework-security/frontend-js/README.md) 和现有案例文档承担相关内容。
|
||||
## 范围元数据
|
||||
|
||||
| 字段 | 内容 |
|
||||
|------|------|
|
||||
| 适用目标类型 | `lab-local`, `lab-public`, `authorized-third-party` |
|
||||
| 是否允许公网验证 | 允许,但只做 Cookie / Storage / Header 边界检查 |
|
||||
| 推荐最小化验证 | 读取响应头、Cookie 属性、DOM 中的存储 API 使用痕迹 |
|
||||
| 禁止场景 | 真实账户会话接管、窃取真实令牌、对第三方浏览器会话做持久利用 |
|
||||
|
||||
## 当前内容
|
||||
|
||||
- 工具: [session-lab.py](/Users/x/websafe/03-authentication/session/tools/session-lab.py)
|
||||
- 关联系统: [frameworks/README.md](/Users/x/websafe/07-framework-security/frameworks/README.md), [platforms/README.md](/Users/x/websafe/07-framework-security/platforms/README.md)
|
||||
- 修复主题: [token-cookie-storage](/Users/x/websafe/05-defense/secure-code/nodejs/token-cookie-storage.md), [proxy-trust-boundary](/Users/x/websafe/05-defense/secure-code/nodejs/proxy-trust-boundary.md)
|
||||
|
||||
@@ -1,5 +1,22 @@
|
||||
# 会话工具说明
|
||||
|
||||
> `LAB NOTE` | `规划中`
|
||||
> `LAB ONLY` | `AUTHORIZED TARGETS ONLY`
|
||||
|
||||
该目录保留给浏览器存储、Cookie 属性和会话边界相关工具。当前尚未补齐具体脚本。
|
||||
## 工具
|
||||
|
||||
- [session-lab.py](/Users/x/websafe/03-authentication/session/tools/session-lab.py)
|
||||
- 用途: Cookie 属性、浏览器存储、可疑认证头与代理边界检查
|
||||
- 目标范围: `lab-local`, `lab-public`, `authorized-third-party`
|
||||
- 允许公网验证: `yes`, 但仅限只读检查
|
||||
- 风险: 低,仅请求目标页面并提取响应头/前端存储痕迹
|
||||
- 不适用: 未授权目标、真实账户会话接管、真实令牌收集
|
||||
|
||||
## 示例
|
||||
|
||||
```bash
|
||||
python3 /Users/x/websafe/03-authentication/session/tools/session-lab.py \
|
||||
--target http://127.0.0.1:18085/ \
|
||||
--ack-authorized \
|
||||
--format json \
|
||||
--evidence-dir /tmp/websafe-session-evidence
|
||||
```
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
- 端口与服务指纹: [port-scanner.py](/Users/x/websafe/04-server-security/scanning/tools/port-scanner.py)
|
||||
- TLS 配置检查: [tls-scanner.py](/Users/x/websafe/04-server-security/tls/tools/tls-scanner.py)
|
||||
- 关联面分析: [site-scope-mapper.py](/Users/x/websafe/04-server-security/infrastructure/tools/site-scope-mapper.py)
|
||||
- 错误配置验证: [misconfig-lab.py](/Users/x/websafe/04-server-security/misconfiguration/tools/misconfig-lab.py)
|
||||
- 实验网关样例: [nginx-hardening.conf](/Users/x/websafe/05-defense/hardening/nginx-hardening.conf)
|
||||
|
||||
## 建议实验路径
|
||||
@@ -23,4 +24,5 @@
|
||||
1. 用 TLS 与响应头检查判断暴露面。
|
||||
2. 用端口扫描确认最小服务面。
|
||||
3. 用关联面分析确认同 IP、同证书和同代理边界。
|
||||
4. 将结果回填到 [资产模板](/Users/x/websafe/09-scope-and-targeting/asset-inventory-template.md) 与 [测试记录模板](/Users/x/websafe/09-scope-and-targeting/test-record-template.md)。
|
||||
4. 对默认页面、调试接口、管理端口和代理信任边界做最小化验证。
|
||||
5. 将结果回填到 [资产模板](/Users/x/websafe/09-scope-and-targeting/asset-inventory-template.md) 与 [测试记录模板](/Users/x/websafe/09-scope-and-targeting/test-record-template.md)。
|
||||
|
||||
@@ -1,5 +1,18 @@
|
||||
# 服务端错误配置实验
|
||||
|
||||
> `LAB NOTE` | `规划中`
|
||||
> `LAB ONLY` | `AUTHORIZED TARGETS ONLY`
|
||||
|
||||
该目录预留给默认目录列表、错误暴露、调试接口、代理信任链和配置合并问题的实验样例。当前相关内容分散在 [07-framework-security/server-software](/Users/x/websafe/07-framework-security/server-software/README.md) 与已有案例中。
|
||||
## 范围元数据
|
||||
|
||||
| 字段 | 内容 |
|
||||
|------|------|
|
||||
| 适用目标类型 | `lab-local`, `lab-public`, `authorized-third-party` |
|
||||
| 是否允许公网验证 | 允许,但必须限定为单站点最小化检查 |
|
||||
| 推荐最小化验证 | 仅访问常见调试、默认页面和健康检查路径 |
|
||||
| 禁止场景 | 大范围爆破目录、未授权管理面探测、影响业务可用性 |
|
||||
|
||||
## 当前内容
|
||||
|
||||
- 工具: [misconfig-lab.py](/Users/x/websafe/04-server-security/misconfiguration/tools/misconfig-lab.py)
|
||||
- 关联系统: [servers/README.md](/Users/x/websafe/07-framework-security/servers/README.md), [platforms/README.md](/Users/x/websafe/07-framework-security/platforms/README.md)
|
||||
- 修复主题: [proxy-trust-boundary](/Users/x/websafe/05-defense/secure-code/nodejs/proxy-trust-boundary.md), [path-traversal-guard](/Users/x/websafe/05-defense/secure-code/nodejs/path-traversal-guard.md)
|
||||
|
||||
@@ -1,5 +1,22 @@
|
||||
# 错误配置工具说明
|
||||
|
||||
> `LAB NOTE` | `规划中`
|
||||
> `LAB ONLY` | `AUTHORIZED TARGETS ONLY`
|
||||
|
||||
该目录后续用于默认配置、目录暴露、调试接口和信任边界误配的辅助检查脚本。
|
||||
## 工具
|
||||
|
||||
- [misconfig-lab.py](/Users/x/websafe/04-server-security/misconfiguration/tools/misconfig-lab.py)
|
||||
- 用途: 默认页面、调试接口、目录暴露、管理端口和危险头部的最小化验证
|
||||
- 目标范围: `lab-local`, `lab-public`, `authorized-third-party`
|
||||
- 允许公网验证: `yes`, 但必须限定到已授权单目标
|
||||
- 风险: 低,只做固定路径的 GET 检查
|
||||
- 不适用: 未授权站点目录枚举、批量互联网画像、DoS/高频探测
|
||||
|
||||
## 示例
|
||||
|
||||
```bash
|
||||
python3 /Users/x/websafe/04-server-security/misconfiguration/tools/misconfig-lab.py \
|
||||
--target http://127.0.0.1:18086/ \
|
||||
--ack-authorized \
|
||||
--format json \
|
||||
--evidence-dir /tmp/websafe-misconfig-evidence
|
||||
```
|
||||
|
||||
@@ -8,16 +8,24 @@
|
||||
|
||||
- [source-map.yaml](/Users/x/websafe/08-threat-intel/source-map.yaml)
|
||||
- 全库唯一真值配置,定义系统范围、覆盖策略、source adapter、输出目录和 secure-code 主题。
|
||||
- [repro-map.yaml](/Users/x/websafe/08-threat-intel/repro-map.yaml)
|
||||
- 系统到 repro family、浏览器要求、seed 策略、日志策略和报告模板的映射。
|
||||
- [repro-profiles/](/Users/x/websafe/08-threat-intel/repro-profiles)
|
||||
- advisory 级和 family 级复现描述,供 `scripts/lab/main.py` 路由执行。
|
||||
- [registry/advisories/](/Users/x/websafe/08-threat-intel/registry/advisories)
|
||||
- canonical advisory 级 JSON 记录,是“所有具体案例”的正式载体。
|
||||
- [registry/systems/](/Users/x/websafe/08-threat-intel/registry/systems)
|
||||
- 每个系统的统计索引、最近更新时间和案例列表。
|
||||
- [registry/runs/](/Users/x/websafe/08-threat-intel/registry/runs)
|
||||
- 每次本地验证的 run bundle 元数据真值,用于反向回填案例页和 dashboard。
|
||||
- [registry/triage/](/Users/x/websafe/08-threat-intel/registry/triage)
|
||||
- 无法自动确定版本、来源冲突或只有弱来源支持的候选。
|
||||
- [generated/coverage-matrix.md](/Users/x/websafe/08-threat-intel/generated/coverage-matrix.md)
|
||||
- 全局覆盖矩阵,展示每个系统的 tier、registry 数、Markdown 数和自动同步状态。
|
||||
- [generated/latest-ingest.md](/Users/x/websafe/08-threat-intel/generated/latest-ingest.md)
|
||||
- 最近一次同步摘要。
|
||||
- [generated/dashboard/](/Users/x/websafe/08-threat-intel/generated/dashboard)
|
||||
- 本地 Web 看板静态产物,展示系统覆盖、状态分布、失败 blocker 和 run bundle 链接。
|
||||
- [registry/source-confidence.md](/Users/x/websafe/08-threat-intel/registry/source-confidence.md)
|
||||
- `official`、`ecosystem-authority`、`research`、`triage-only` 的入库规则。
|
||||
|
||||
@@ -51,6 +59,11 @@ python3 /Users/x/websafe/scripts/intel/main.py reconcile
|
||||
python3 /Users/x/websafe/scripts/intel/main.py backfill --tier rolling-24m --dry-run
|
||||
python3 /Users/x/websafe/scripts/intel/main.py ingest --since 365d --system nextjs --system vite
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr --dry-run
|
||||
python3 /Users/x/websafe/scripts/lab/main.py validate
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-case --case gitea--CVE-2025-68939
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-case --case nextjs--CVE-2025-29927 --dry-run
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-batch --only-hotlane --limit 10
|
||||
python3 /Users/x/websafe/scripts/lab/main.py serve-dashboard --port 8734
|
||||
```
|
||||
|
||||
可选环境变量:
|
||||
@@ -63,6 +76,7 @@ python3 /Users/x/websafe/scripts/intel/main.py open-pr --dry-run
|
||||
运行时建议:
|
||||
|
||||
- 使用独立虚拟环境安装 [requirements-intel.txt](/Users/x/websafe/requirements-intel.txt)。
|
||||
- 浏览器类 case 需要执行 `python3 -m playwright install chromium` 安装浏览器运行时。
|
||||
- 当前机器上的 Python 3.9 + LibreSSL 对部分 HTTPS 源可能出现 `SSLError`;`urllib3<2` 已写入依赖约束以降低兼容性问题。
|
||||
|
||||
对应的本机 cron 入口:
|
||||
@@ -81,9 +95,12 @@ python3 /Users/x/websafe/scripts/intel/main.py open-pr --dry-run
|
||||
- [intake-and-severity-rules.md](/Users/x/websafe/08-threat-intel/intake-and-severity-rules.md)
|
||||
- [case-intake-template.md](/Users/x/websafe/08-threat-intel/case-intake-template.md)
|
||||
- [config-examples/README.md](/Users/x/websafe/08-threat-intel/config-examples/README.md)
|
||||
- [generated/dashboard/index.html](/Users/x/websafe/08-threat-intel/generated/dashboard/index.html)
|
||||
- [registry/runs](/Users/x/websafe/08-threat-intel/registry/runs)
|
||||
|
||||
## 实验边界
|
||||
|
||||
- 所有案例、source adapter 和索引页仅适用于 `lab-local`、`lab-public`、`authorized-third-party`。
|
||||
- 允许公网可达目标,但前提必须是资产归属明确,或已获得明确授权。
|
||||
- 不面向未授权互联网资产,不面向公共知名网站,不作为泛化枚举或生产推荐语境。
|
||||
- 前端 / 浏览器类 advisory 若没有浏览器层回放与证据,只能停留在 `triage-manual` 或 `suspected`,不得宣称完成实证。
|
||||
|
||||
50
README.md
50
README.md
@@ -17,30 +17,31 @@
|
||||
|
||||
```text
|
||||
websafe/
|
||||
├── 00-environments/ # 靶场与本地实验编排
|
||||
├── 00-environments/ # 系统 catalog、真实版本/当前版本 profile、synthetic 模板
|
||||
├── 01-sql-injection/ # SQL 注入实验
|
||||
├── 02-xss/ # XSS 与浏览器端注入实验
|
||||
├── 03-authentication/ # 认证、会话与 JWT 实验
|
||||
├── 04-server-security/ # 服务器、TLS、暴露面与关联面实验
|
||||
├── 05-defense/ # 检测、观测、实验对照与代码修复示例
|
||||
├── 06-case-studies/ # 授权案例与原始报告归档
|
||||
├── 06-case-studies/ # 授权案例与 run bundle / 报告归档
|
||||
├── 07-framework-security/ # CMS、电商、框架、服务器、平台系统安全
|
||||
├── 08-threat-intel/ # source-map、registry、generated、订阅规则、自动入库
|
||||
├── 08-threat-intel/ # source-map、repro-map、registry、dashboard、订阅规则、自动入库
|
||||
├── 09-scope-and-targeting/ # 授权模型、资产模板、测试记录模板
|
||||
├── requirements-intel.txt # intel 自动化依赖
|
||||
└── scripts/intel/ # hotlane / ingest / reconcile / backfill / open-pr CLI
|
||||
├── requirements-intel.txt # intel + lab 自动化依赖(含 Playwright Python 包)
|
||||
├── scripts/intel/ # hotlane / ingest / reconcile / backfill / open-pr CLI
|
||||
└── scripts/lab/ # provision / baseline / attack / browser / evidence / render / queue CLI
|
||||
```
|
||||
|
||||
## 能力矩阵
|
||||
|
||||
| 覆盖域 | 历史全量策略 | 近两年策略 | 全量 registry | 重点案例 Markdown | secure-code 关联 | 自动同步状态 |
|
||||
|--------|--------------|------------|---------------|--------------------|------------------|--------------|
|
||||
| CMS / 内容平台 | `WordPress`, `Drupal`, `Joomla` | `Ghost`, `Strapi`, `Directus`, `MediaWiki`, `Moodle`, `Discourse` | `registry/advisories + registry/systems` | `core 全量 + 高价值 extension` | `yes` | `render / ingest / hotlane / reconcile ready` |
|
||||
| 电商系统 | `Adobe Commerce`, `Magento Open Source`, `WooCommerce`, `PrestaShop`, `Shopware`, `OpenCart` | `OpenMage`, `Saleor`, `Medusa` | `registry/advisories + registry/systems` | `core 全量 + 高价值 module` | `yes` | `render / ingest / hotlane / reconcile ready` |
|
||||
| Web 框架与运行时 | `React`, `Next.js`, `Vue`, `Nuxt`, `Vite`, `Node.js`, `Nginx`, `Apache HTTP Server`, `Apache Tomcat` | 其余主流框架与运行时按 `rolling-24m` | `registry/advisories + registry/systems` | `core 全量 + 高价值 package` | `yes` | `render / ingest / hotlane / reconcile ready` |
|
||||
| 开源平台与后台系统 | `history-full` 不强制 | `phpMyAdmin`, `Adminer`, `Gitea`, `GitLab CE`, `Jenkins`, `Grafana`, `Kibana`, `Mattermost`, `Redmine` | `registry/advisories + registry/systems` | `高价值案例输出` | `yes` | `render / ingest / hotlane / reconcile ready` |
|
||||
| 修复示例库 | 不适用 | 不适用 | 不适用 | 由案例页反向链接 | `javascript-typescript`, `nodejs`, `java`, `php`, `python`, `ruby`, `csharp`, `go` | `render ready` |
|
||||
| 自动化入库 | `backfill --tier history-full` | `ingest --since`, `reconcile` | `registry + generated` | `基于 render_policy` | `front matter 反向链接` | `open-pr dry-run ready` |
|
||||
| 覆盖域 | 历史全量策略 | 近两年策略 | 全量 registry | 重点案例 Markdown | secure-code 关联 | 本地实证状态 | 浏览器证据 | run bundle | 看板展示 | 自动同步状态 |
|
||||
|--------|--------------|------------|---------------|--------------------|------------------|--------------|------------|-----------|----------|--------------|
|
||||
| CMS / 内容平台 | `WordPress`, `Drupal`, `Joomla` | `Ghost`, `Strapi`, `Directus`, `MediaWiki`, `Moodle`, `Discourse` | `registry/advisories + registry/systems` | `core 全量 + 高价值 extension` | `yes` | `verified-real / verified-synthetic / blocked-* / triage-manual` | `前端类强制` | `06-case-studies/generated-runs` | `dashboard + report` | `render / ingest / hotlane / reconcile ready` |
|
||||
| 电商系统 | `Adobe Commerce`, `Magento Open Source`, `WooCommerce`, `PrestaShop`, `Shopware`, `OpenCart` | `OpenMage`, `Saleor`, `Medusa` | `registry/advisories + registry/systems` | `core 全量 + 高价值 module` | `yes` | `同上` | `前台/后台面板类强制` | `run bundle + logs` | `dashboard + report` | `render / ingest / hotlane / reconcile ready` |
|
||||
| Web 框架与运行时 | `React`, `Next.js`, `Vue`, `Nuxt`, `Vite`, `Node.js`, `Nginx`, `Apache HTTP Server`, `Apache Tomcat` | 其余主流框架与运行时按 `rolling-24m` | `registry/advisories + registry/systems` | `core 全量 + 高价值 package` | `yes` | `family runner + advisory profile` | `浏览器/HTTP 混合` | `run bundle + timeline` | `dashboard + report` | `render / ingest / hotlane / reconcile ready` |
|
||||
| 开源平台与后台系统 | `history-full` 不强制 | `phpMyAdmin`, `Adminer`, `Gitea`, `GitLab CE`, `Jenkins`, `Grafana`, `Kibana`, `Mattermost`, `Redmine` | `registry/advisories + registry/systems` | `高价值案例输出` | `yes` | `真实版本优先` | `Web 面板类强制` | `run bundle + screenshots` | `dashboard + report` | `render / ingest / hotlane / reconcile ready` |
|
||||
| 修复示例库 | 不适用 | 不适用 | 不适用 | 由案例页反向链接 | `javascript-typescript`, `nodejs`, `java`, `php`, `python`, `ruby`, `csharp`, `go` | `由案例反向映射` | `不适用` | `不适用` | `索引页` | `render ready` |
|
||||
| 自动化入库与实证 | `backfill --tier history-full` | `ingest --since`, `reconcile` | `registry + generated + registry/runs` | `基于 render_policy` | `front matter 反向链接` | `queue + run-case / run-batch` | `Playwright required for browser cases` | `report.md / report.html / timeline.mmd` | `serve-dashboard` | `open-pr / cron ready` |
|
||||
|
||||
## 当前覆盖对象
|
||||
|
||||
@@ -62,13 +63,30 @@ python3 /Users/x/websafe/scripts/intel/main.py ingest --since last-success
|
||||
python3 /Users/x/websafe/scripts/intel/main.py reconcile
|
||||
python3 /Users/x/websafe/scripts/intel/main.py backfill --tier history-full --dry-run
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr --dry-run
|
||||
python3 /Users/x/websafe/scripts/lab/main.py catalog sync
|
||||
python3 /Users/x/websafe/scripts/lab/main.py validate
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-case --case nextjs--CVE-2025-29927 --dry-run
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-batch --only-hotlane --limit 10
|
||||
python3 /Users/x/websafe/scripts/lab/main.py serve-dashboard --port 8734
|
||||
```
|
||||
|
||||
计划中的本机 cron 入口:
|
||||
|
||||
- [run-hourly.sh](/Users/x/websafe/scripts/intel/run-hourly.sh) 只处理 KEV / 在野利用 / 极高优先级更新
|
||||
- [run-nightly.sh](/Users/x/websafe/scripts/intel/run-nightly.sh) 处理常规增量同步
|
||||
- [run-weekly-reconcile.sh](/Users/x/websafe/scripts/intel/run-weekly-reconcile.sh) 对齐最近 30 天更新
|
||||
- [run-hourly.sh](/Users/x/websafe/scripts/intel/run-hourly.sh) 处理 KEV / 在野利用 / 极高优先级更新,并触发 hotlane 实证队列
|
||||
- [run-nightly.sh](/Users/x/websafe/scripts/intel/run-nightly.sh) 处理常规增量同步、批量实证、dashboard 渲染和 PR
|
||||
- [run-weekly-reconcile.sh](/Users/x/websafe/scripts/intel/run-weekly-reconcile.sh) 对齐最近 30 天更新,并重跑失败/阻塞任务
|
||||
|
||||
## 本地实证链路
|
||||
|
||||
每条 advisory 的自动链路固定为:
|
||||
|
||||
1. `registry/advisories/*.json` 选中 case。
|
||||
2. `repro-map.yaml + repro-profiles/` 解析到 repro family / advisory profile。
|
||||
3. `00-environments/catalog + profiles` 生成 compose 拓扑和靶站参数。
|
||||
4. `scripts/lab/main.py run-case` 拉起环境、收集 baseline、执行受控攻击链。
|
||||
5. 前端类 case 强制走 Playwright 浏览器回放,生成截图、DOM、console、network 证据。
|
||||
6. 生成 `06-case-studies/generated-runs/<run-id>/` 报告和 `08-threat-intel/registry/runs/<run-id>.json`。
|
||||
7. 自动回写 registry、系统 INDEX、案例页和 dashboard。
|
||||
|
||||
## 实验边界
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import os
|
||||
import re
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
from urllib.parse import quote
|
||||
|
||||
import requests
|
||||
|
||||
@@ -13,16 +14,24 @@ from intel.utils import read_json, run
|
||||
|
||||
PR_PATHS = [
|
||||
"README.md",
|
||||
"00-environments",
|
||||
"01-sql-injection",
|
||||
"02-xss",
|
||||
"03-authentication",
|
||||
"04-server-security",
|
||||
"05-defense/secure-code",
|
||||
"06-case-studies/generated-runs",
|
||||
"07-framework-security",
|
||||
"08-threat-intel",
|
||||
"requirements-intel.txt",
|
||||
"scripts/intel",
|
||||
"scripts/lab",
|
||||
"scripts/tool_contract.py",
|
||||
]
|
||||
|
||||
|
||||
def create_branch_name() -> str:
|
||||
return "codex/intel-" + datetime.now().strftime("%Y%m%d-%H%M")
|
||||
return "codex/intel-" + datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
|
||||
|
||||
def _parse_origin() -> Optional[dict]:
|
||||
@@ -42,6 +51,17 @@ def _changed_paths() -> list[str]:
|
||||
return lines
|
||||
|
||||
|
||||
def _current_branch() -> str:
|
||||
result = run(["git", "-C", str(ROOT), "branch", "--show-current"], check=False)
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def _push_remote(origin: dict, token: str | None) -> str:
|
||||
if token and origin["url"].startswith("https://"):
|
||||
return f"https://{quote(origin['owner'], safe='')}:{quote(token, safe='')}@{origin['host']}/{origin['owner']}/{origin['repo']}.git"
|
||||
return "origin"
|
||||
|
||||
|
||||
def open_pr(base_branch: str = "main", dry_run: bool = False) -> str:
|
||||
origin = _parse_origin()
|
||||
if not origin:
|
||||
@@ -51,27 +71,36 @@ def open_pr(base_branch: str = "main", dry_run: bool = False) -> str:
|
||||
if not changed:
|
||||
return "No intel-related changes to submit"
|
||||
|
||||
branch = _current_branch()
|
||||
if not branch.startswith("codex/"):
|
||||
branch = create_branch_name()
|
||||
if dry_run:
|
||||
preview = "\n".join(f"- {line}" for line in changed[:40])
|
||||
return f"Dry run only; would create branch {branch} with these paths:\n{preview}"
|
||||
|
||||
if _current_branch() != branch:
|
||||
run(["git", "-C", str(ROOT), "checkout", "-b", branch])
|
||||
run(["git", "-C", str(ROOT), "add", "--", *PR_PATHS])
|
||||
run(["git", "-C", str(ROOT), "commit", "-m", f"intel: automated advisory ingest {branch}"])
|
||||
run(["git", "-C", str(ROOT), "push", "-u", "origin", branch])
|
||||
run(["git", "-C", str(ROOT), "commit", "-m", f"lab: automated intel and verification sync {branch}"])
|
||||
|
||||
token = os.environ.get("GITEA_TOKEN")
|
||||
run(["git", "-C", str(ROOT), "push", "-u", _push_remote(origin, token), branch])
|
||||
if not token:
|
||||
return f"Pushed branch {branch}, but GITEA_TOKEN is not set; PR not created"
|
||||
|
||||
summary = read_json(GENERATED_DIR / "run-summary.json", default={}) or {}
|
||||
dashboard = read_json(GENERATED_DIR / "dashboard" / "summary.json", default={}) or {}
|
||||
body_lines = [
|
||||
"Automated advisory ingest update.",
|
||||
"Automated advisory ingest and local verification update.",
|
||||
"",
|
||||
f"- New advisories: {summary.get('new_count', 0)}",
|
||||
f"- Updated advisories: {summary.get('updated_count', 0)}",
|
||||
f"- Triage count: {summary.get('triage_count', 0)}",
|
||||
f"- Run bundles: {summary.get('run_bundle_count', 0)}",
|
||||
f"- verified-real: {dashboard.get('statuses', {}).get('verified-real', 0)}",
|
||||
f"- verified-synthetic: {dashboard.get('statuses', {}).get('verified-synthetic', 0)}",
|
||||
f"- blocked-artifact: {dashboard.get('statuses', {}).get('blocked-artifact', 0)}",
|
||||
f"- triage-manual: {dashboard.get('statuses', {}).get('triage-manual', 0)}",
|
||||
f"- Failure count: {len(summary.get('failures', []))}",
|
||||
]
|
||||
if summary.get("systems_touched"):
|
||||
@@ -80,6 +109,10 @@ def open_pr(base_branch: str = "main", dry_run: bool = False) -> str:
|
||||
body_lines.extend(["", "Failed source adapters:"])
|
||||
for failure in summary["failures"]:
|
||||
body_lines.append(f"- {failure}")
|
||||
if dashboard.get("recent_failures"):
|
||||
body_lines.extend(["", "Recent repro blockers:"])
|
||||
for failure in dashboard["recent_failures"][:10]:
|
||||
body_lines.append(f"- {failure['run_id']} :: {failure['status']} :: {failure.get('blocked_reason') or '-'}")
|
||||
|
||||
payload = {
|
||||
"title": f"Intel ingest {branch}",
|
||||
|
||||
@@ -7,5 +7,19 @@ mkdir -p "$LOG_DIR"
|
||||
STAMP="$(date '+%Y%m%d-%H%M%S')"
|
||||
exec >> "$LOG_DIR/hourly-$STAMP.log" 2>&1
|
||||
|
||||
run_pr() {
|
||||
if [[ "${WEBSAFE_PR_MODE:-auto}" == "skip" ]]; then
|
||||
echo "[hourly] PR skipped by WEBSAFE_PR_MODE=skip"
|
||||
elif [[ "${WEBSAFE_PR_MODE:-auto}" == "dry-run" || -z "${GITEA_TOKEN:-}" ]]; then
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr --dry-run
|
||||
else
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr
|
||||
fi
|
||||
}
|
||||
|
||||
echo "[hourly] $(date -u '+%Y-%m-%dT%H:%M:%SZ') starting"
|
||||
python3 /Users/x/websafe/scripts/intel/main.py hotlane
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-batch --only-hotlane --limit "${WEBSAFE_HOTLANE_LIMIT:-10}"
|
||||
python3 /Users/x/websafe/scripts/intel/main.py render
|
||||
python3 /Users/x/websafe/scripts/intel/main.py validate
|
||||
run_pr
|
||||
|
||||
@@ -7,5 +7,19 @@ mkdir -p "$LOG_DIR"
|
||||
STAMP="$(date '+%Y%m%d-%H%M%S')"
|
||||
exec >> "$LOG_DIR/nightly-$STAMP.log" 2>&1
|
||||
|
||||
run_pr() {
|
||||
if [[ "${WEBSAFE_PR_MODE:-auto}" == "skip" ]]; then
|
||||
echo "[nightly] PR skipped by WEBSAFE_PR_MODE=skip"
|
||||
elif [[ "${WEBSAFE_PR_MODE:-auto}" == "dry-run" || -z "${GITEA_TOKEN:-}" ]]; then
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr --dry-run
|
||||
else
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr
|
||||
fi
|
||||
}
|
||||
|
||||
echo "[nightly] $(date -u '+%Y-%m-%dT%H:%M:%SZ') starting"
|
||||
python3 /Users/x/websafe/scripts/intel/main.py ingest --since last-success
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-batch --limit "${WEBSAFE_NIGHTLY_LIMIT:-25}"
|
||||
python3 /Users/x/websafe/scripts/intel/main.py render
|
||||
python3 /Users/x/websafe/scripts/intel/main.py validate
|
||||
run_pr
|
||||
|
||||
@@ -7,5 +7,20 @@ mkdir -p "$LOG_DIR"
|
||||
STAMP="$(date '+%Y%m%d-%H%M%S')"
|
||||
exec >> "$LOG_DIR/weekly-$STAMP.log" 2>&1
|
||||
|
||||
run_pr() {
|
||||
if [[ "${WEBSAFE_PR_MODE:-auto}" == "skip" ]]; then
|
||||
echo "[weekly] PR skipped by WEBSAFE_PR_MODE=skip"
|
||||
elif [[ "${WEBSAFE_PR_MODE:-auto}" == "dry-run" || -z "${GITEA_TOKEN:-}" ]]; then
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr --dry-run
|
||||
else
|
||||
python3 /Users/x/websafe/scripts/intel/main.py open-pr
|
||||
fi
|
||||
}
|
||||
|
||||
echo "[weekly] $(date -u '+%Y-%m-%dT%H:%M:%SZ') starting"
|
||||
python3 /Users/x/websafe/scripts/intel/main.py reconcile
|
||||
python3 /Users/x/websafe/scripts/lab/main.py retry-failures --limit "${WEBSAFE_RETRY_LIMIT:-100}"
|
||||
python3 /Users/x/websafe/scripts/lab/main.py run-batch --from-queue --limit "${WEBSAFE_WEEKLY_LIMIT:-50}"
|
||||
python3 /Users/x/websafe/scripts/intel/main.py render
|
||||
python3 /Users/x/websafe/scripts/intel/main.py validate
|
||||
run_pr
|
||||
|
||||
@@ -83,6 +83,7 @@ def validate(source_map: Dict[str, Any]) -> List[str]:
|
||||
GENERATED_DIR / "run-summary.json",
|
||||
GENERATED_DIR / "dashboard" / "index.html",
|
||||
GENERATED_DIR / "dashboard" / "summary.json",
|
||||
GENERATED_DIR / "dashboard" / "systems.json",
|
||||
ROOT / "08-threat-intel" / "registry" / "source-confidence.md",
|
||||
]:
|
||||
if not path.exists():
|
||||
|
||||
@@ -25,8 +25,12 @@ def capture(url: str, run_dir: Path, prefix: str = "baseline") -> Dict[str, Any]
|
||||
dom_path = assets_dir / f"{prefix}-dom.html"
|
||||
console_path = run_dir / "logs" / f"{prefix}-console.json"
|
||||
network_path = run_dir / "logs" / f"{prefix}-network.json"
|
||||
page_path = run_dir / "logs" / f"{prefix}-page.json"
|
||||
console_messages: List[Dict[str, Any]] = []
|
||||
requests_seen: List[Dict[str, Any]] = []
|
||||
page_title = ""
|
||||
page_body_excerpt = ""
|
||||
final_url = url
|
||||
try:
|
||||
with sync_playwright() as p:
|
||||
browser = p.chromium.launch(headless=True)
|
||||
@@ -36,6 +40,9 @@ def capture(url: str, run_dir: Path, prefix: str = "baseline") -> Dict[str, Any]
|
||||
page.goto(url, wait_until="networkidle", timeout=20000)
|
||||
page.screenshot(path=str(screenshot_path), full_page=True)
|
||||
dom_path.write_text(page.content(), encoding="utf-8")
|
||||
final_url = page.url
|
||||
page_title = page.title()
|
||||
page_body_excerpt = (page.text_content("body") or "")[:600]
|
||||
browser.close()
|
||||
except Exception as exc:
|
||||
payload["reason"] = str(exc)
|
||||
@@ -43,10 +50,20 @@ def capture(url: str, run_dir: Path, prefix: str = "baseline") -> Dict[str, Any]
|
||||
return payload
|
||||
write_json(console_path, console_messages)
|
||||
write_json(network_path, requests_seen)
|
||||
write_json(
|
||||
page_path,
|
||||
{
|
||||
"url": final_url,
|
||||
"title": page_title,
|
||||
"body_excerpt": page_body_excerpt,
|
||||
},
|
||||
)
|
||||
payload = {
|
||||
"required": True,
|
||||
"present": True,
|
||||
"refs": [str(screenshot_path), str(dom_path), str(console_path), str(network_path)],
|
||||
"page_title": page_title,
|
||||
"page_url": final_url,
|
||||
"refs": [str(screenshot_path), str(dom_path), str(console_path), str(network_path), str(page_path)],
|
||||
}
|
||||
write_json(run_dir / "logs" / f"{prefix}-browser.json", payload)
|
||||
return payload
|
||||
|
||||
@@ -35,6 +35,32 @@ def _compose_run_id(advisory: Dict[str, Any]) -> str:
|
||||
return f"{advisory['system_id']}-{advisory['canonical_id']}-{now_utc().strftime('%Y%m%d%H%M%S')}"
|
||||
|
||||
|
||||
def _existing_refs(*paths: Path) -> List[str]:
|
||||
return [str(path) for path in paths if path.exists()]
|
||||
|
||||
|
||||
def _timeline_event(timeline: List[Dict[str, Any]], step: str, status: str, detail: str = "") -> None:
|
||||
timeline.append(
|
||||
{
|
||||
"at": isoformat(now_utc()),
|
||||
"step": step,
|
||||
"status": status,
|
||||
"detail": detail,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _sync_registry_outputs() -> None:
|
||||
from intel.config import GENERATED_DIR, load_source_map # noqa: E402
|
||||
from intel.main import _load_existing_advisories, _load_existing_triage, _write_outputs # noqa: E402
|
||||
|
||||
source_map = load_source_map()
|
||||
advisories = _load_existing_advisories()
|
||||
triage = _load_existing_triage()
|
||||
summary = read_json(GENERATED_DIR / "run-summary.json", default={}) or {}
|
||||
_write_outputs(source_map, advisories, triage, summary.get("failures", []), summary)
|
||||
|
||||
|
||||
def _resolve_profile(advisory: Dict[str, Any]) -> Dict[str, Any]:
|
||||
profile = repro.resolve_profile(advisory["canonical_id"], advisory)
|
||||
current_profile = read_yaml(ENV_PROFILES_DIR / "core" / advisory["system_id"] / "current.yaml", default={}) or {}
|
||||
@@ -70,6 +96,11 @@ def _build_run_bundle(
|
||||
browser_refs: List[str],
|
||||
container_log_refs: List[str],
|
||||
request_log_refs: List[str],
|
||||
compose_refs: List[str],
|
||||
browser_evidence: Dict[str, Any],
|
||||
timeline: List[Dict[str, Any]],
|
||||
started_at: str,
|
||||
finished_at: str,
|
||||
blocked_reason: str | None,
|
||||
) -> Dict[str, Any]:
|
||||
return {
|
||||
@@ -85,15 +116,174 @@ def _build_run_bundle(
|
||||
"baseline_refs": baseline_refs,
|
||||
"attack_steps": attack_steps,
|
||||
"browser_refs": browser_refs,
|
||||
"browser_evidence": browser_evidence,
|
||||
"container_log_refs": container_log_refs,
|
||||
"request_log_refs": request_log_refs,
|
||||
"timeline": [],
|
||||
"started_at": isoformat(now_utc()),
|
||||
"finished_at": isoformat(now_utc()),
|
||||
"compose_refs": compose_refs,
|
||||
"timeline": timeline,
|
||||
"started_at": started_at,
|
||||
"finished_at": finished_at,
|
||||
"blocked_reason": blocked_reason,
|
||||
}
|
||||
|
||||
|
||||
def _dry_run_case_plan(advisory: Dict[str, Any], profile: Dict[str, Any], run_id: str) -> Dict[str, Any]:
|
||||
provision_result = provision.prepare(profile, CASE_RUNS_DIR / run_id, dry_run=True)
|
||||
return {
|
||||
"run_id": run_id,
|
||||
"system_id": advisory["system_id"],
|
||||
"advisory_id": advisory["canonical_id"],
|
||||
"repro_profile_id": profile["profile_id"],
|
||||
"verification_mode": profile.get("verification_mode", "synthetic"),
|
||||
"artifact_mode": profile.get("artifact_mode", profile.get("provisioning_mode", "synthetic")),
|
||||
"browser_required": bool(profile.get("browser_assertions", {}).get("required")),
|
||||
"baseline_urls": profile.get("baseline_urls", []),
|
||||
"compose_services": sorted(profile.get("services", {}).keys()),
|
||||
"seed_actions": profile.get("seed_actions", []),
|
||||
"attack_actions": profile.get("attack_actions", []),
|
||||
"compose_preview": provision_result.get("compose_preview", {}),
|
||||
"note": "dry-run only; no bundle, report, compose file, or registry update was written",
|
||||
}
|
||||
|
||||
|
||||
def _execute_case(canonical_id: str, run_id: str | None = None, dry_run: bool = False, sync_outputs: bool = True) -> Dict[str, Any]:
|
||||
advisory = _load_advisory(canonical_id)
|
||||
profile = _resolve_profile(advisory)
|
||||
resolved_run_id = run_id or _compose_run_id(advisory)
|
||||
if dry_run:
|
||||
return _dry_run_case_plan(advisory, profile, resolved_run_id)
|
||||
|
||||
started_at = isoformat(now_utc())
|
||||
timeline: List[Dict[str, Any]] = []
|
||||
_timeline_event(timeline, "select-advisory", "completed", advisory["canonical_id"])
|
||||
_timeline_event(timeline, "resolve-repro-profile", "completed", profile["profile_id"])
|
||||
|
||||
run_dir = _run_dir(resolved_run_id)
|
||||
provision_result = provision.prepare(profile, run_dir, dry_run=False)
|
||||
_timeline_event(
|
||||
timeline,
|
||||
"provision-compose-environment",
|
||||
provision_result.get("status", "unknown"),
|
||||
provision_result.get("blocked_reason", ""),
|
||||
)
|
||||
allow_runtime_steps = provision_result.get("status") not in {"blocked-artifact"}
|
||||
browser_required = bool(profile.get("browser_assertions", {}).get("required"))
|
||||
|
||||
baseline_payload = {"observations": []}
|
||||
if profile.get("baseline_urls") and allow_runtime_steps:
|
||||
baseline_payload = baseline.collect(profile, run_dir)
|
||||
_timeline_event(timeline, "baseline-snapshot", "completed", f"urls={len(profile.get('baseline_urls', []))}")
|
||||
else:
|
||||
_timeline_event(timeline, "baseline-snapshot", "skipped", "no baseline urls or provisioning blocked")
|
||||
|
||||
baseline_browser = {"required": browser_required, "present": False, "refs": []}
|
||||
if browser_required and allow_runtime_steps and profile.get("baseline_urls"):
|
||||
baseline_browser = browser.capture(profile["baseline_urls"][0], run_dir, prefix="baseline")
|
||||
_timeline_event(
|
||||
timeline,
|
||||
"browser-replay-before-attack",
|
||||
"completed" if baseline_browser.get("present") else "failed",
|
||||
baseline_browser.get("reason", ""),
|
||||
)
|
||||
elif browser_required:
|
||||
_timeline_event(timeline, "browser-replay-before-attack", "skipped", "baseline browser capture unavailable")
|
||||
|
||||
attack_payload = {"steps": []}
|
||||
if allow_runtime_steps:
|
||||
attack_payload = attack.run_attack(profile, advisory, run_dir, dry_run=False)
|
||||
attack_failed = any(step.get("status") == "failed" for step in attack_payload.get("steps", []))
|
||||
_timeline_event(
|
||||
timeline,
|
||||
"controlled-attack-chain",
|
||||
"failed" if attack_failed else "completed",
|
||||
f"steps={len(attack_payload.get('steps', []))}",
|
||||
)
|
||||
else:
|
||||
_timeline_event(timeline, "controlled-attack-chain", "skipped", "provisioning blocked")
|
||||
|
||||
proof_browser = {"required": browser_required, "present": False, "refs": []}
|
||||
if browser_required and allow_runtime_steps and profile.get("baseline_urls"):
|
||||
proof_browser = browser.capture(profile["baseline_urls"][0], run_dir, prefix="proof")
|
||||
_timeline_event(
|
||||
timeline,
|
||||
"browser-replay-after-attack",
|
||||
"completed" if proof_browser.get("present") else "failed",
|
||||
proof_browser.get("reason", ""),
|
||||
)
|
||||
elif browser_required:
|
||||
_timeline_event(timeline, "browser-replay-after-attack", "skipped", "proof browser capture unavailable")
|
||||
|
||||
compose_path = Path(provision_result["compose_path"])
|
||||
container_logs = evidence.collect_container_logs(run_dir, compose_path) if compose_path.exists() and allow_runtime_steps else []
|
||||
_timeline_event(
|
||||
timeline,
|
||||
"collect-logs-and-evidence",
|
||||
"completed" if allow_runtime_steps else "skipped",
|
||||
f"container_logs={len(container_logs)}",
|
||||
)
|
||||
|
||||
browser_present = bool(baseline_browser.get("present")) and bool(proof_browser.get("present"))
|
||||
browser_payload = {
|
||||
"required": browser_required,
|
||||
"present": browser_present,
|
||||
"refs": baseline_browser.get("refs", []) + proof_browser.get("refs", []),
|
||||
"baseline_refs": baseline_browser.get("refs", []),
|
||||
"proof_refs": proof_browser.get("refs", []),
|
||||
"baseline_title": baseline_browser.get("page_title"),
|
||||
"proof_title": proof_browser.get("page_title"),
|
||||
}
|
||||
|
||||
blocked_reason = provision_result.get("blocked_reason")
|
||||
if browser_required and not browser_present:
|
||||
blocked_reason = blocked_reason or baseline_browser.get("reason") or proof_browser.get("reason") or "browser evidence incomplete"
|
||||
|
||||
verification_mode = profile.get("verification_mode", "synthetic")
|
||||
artifact_mode = profile.get("artifact_mode", profile.get("provisioning_mode", "synthetic"))
|
||||
verification_status = "triage-manual"
|
||||
if provision_result.get("status") == "blocked-artifact":
|
||||
verification_status = "blocked-artifact"
|
||||
elif browser_required and not browser_present:
|
||||
verification_status = "triage-manual"
|
||||
elif any(step.get("status") == "failed" for step in attack_payload.get("steps", [])):
|
||||
verification_status = "triage-manual"
|
||||
elif artifact_mode == "synthetic":
|
||||
verification_status = "verified-synthetic"
|
||||
else:
|
||||
verification_status = "verified-real"
|
||||
|
||||
finished_at = isoformat(now_utc())
|
||||
bundle = _build_run_bundle(
|
||||
advisory=advisory,
|
||||
profile=profile,
|
||||
run_id=resolved_run_id,
|
||||
verification_status=verification_status,
|
||||
verification_mode=verification_mode,
|
||||
artifact_mode=artifact_mode,
|
||||
baseline_refs=_existing_refs(run_dir / "logs" / "baseline.json"),
|
||||
attack_steps=attack_payload.get("steps", []),
|
||||
browser_refs=browser_payload["refs"],
|
||||
container_log_refs=container_logs,
|
||||
request_log_refs=_existing_refs(run_dir / "logs" / "attack.json", run_dir / "logs" / "baseline.json"),
|
||||
compose_refs=[str(compose_path)] if compose_path.exists() else [],
|
||||
browser_evidence=browser_payload,
|
||||
timeline=timeline,
|
||||
started_at=started_at,
|
||||
finished_at=finished_at,
|
||||
blocked_reason=blocked_reason,
|
||||
)
|
||||
_timeline_event(bundle["timeline"], "update-registry-and-reports", "completed", resolved_run_id)
|
||||
report_refs = render.render_run(bundle)
|
||||
bundle["report_refs"] = report_refs
|
||||
evidence.write_run_bundle(run_dir, bundle)
|
||||
ensure_dir(RUNS_DIR)
|
||||
write_json(RUNS_DIR / f"{resolved_run_id}.json", bundle)
|
||||
if sync_outputs:
|
||||
_sync_registry_outputs()
|
||||
else:
|
||||
render.render_dashboard()
|
||||
return bundle
|
||||
|
||||
|
||||
def cmd_catalog_sync(args) -> int:
|
||||
summary = catalog.sync_catalog(write_profiles=True, write_repro_map=True)
|
||||
print(summary)
|
||||
@@ -158,66 +348,8 @@ def cmd_verify(args) -> int:
|
||||
|
||||
|
||||
def cmd_run_case(args) -> int:
|
||||
advisory = _load_advisory(args.case)
|
||||
profile = _resolve_profile(advisory)
|
||||
run_id = args.run_id or _compose_run_id(advisory)
|
||||
run_dir = _run_dir(run_id)
|
||||
|
||||
provision_result = provision.prepare(profile, run_dir, dry_run=args.dry_run)
|
||||
allow_runtime_steps = provision_result.get("status") not in {"blocked-artifact"}
|
||||
baseline_payload = (
|
||||
baseline.collect(profile, run_dir) if profile.get("baseline_urls") and allow_runtime_steps else {"observations": []}
|
||||
)
|
||||
attack_payload = (
|
||||
attack.run_attack(profile, advisory, run_dir, dry_run=args.dry_run) if allow_runtime_steps else {"steps": []}
|
||||
)
|
||||
|
||||
browser_payload = {"required": bool(profile.get("browser_assertions", {}).get("required")), "present": False, "refs": []}
|
||||
blocked_reason = provision_result.get("blocked_reason")
|
||||
if browser_payload["required"] and not args.dry_run and profile.get("baseline_urls") and allow_runtime_steps:
|
||||
browser_payload = browser.capture(profile["baseline_urls"][0], run_dir, prefix="proof")
|
||||
if not browser_payload.get("present"):
|
||||
blocked_reason = blocked_reason or browser_payload.get("reason")
|
||||
|
||||
compose_path = Path(provision_result["compose_path"])
|
||||
container_logs = evidence.collect_container_logs(run_dir, compose_path) if compose_path.exists() and allow_runtime_steps else []
|
||||
|
||||
verification_status = "triage-manual"
|
||||
verification_mode = profile.get("verification_mode", "synthetic")
|
||||
artifact_mode = profile.get("artifact_mode", profile.get("provisioning_mode", "synthetic"))
|
||||
if args.dry_run:
|
||||
verification_status = "triage-manual"
|
||||
blocked_reason = blocked_reason or "dry-run only"
|
||||
elif provision_result.get("status") == "blocked-artifact":
|
||||
verification_status = "blocked-artifact"
|
||||
elif browser_payload.get("required") and not browser_payload.get("present"):
|
||||
verification_status = "triage-manual"
|
||||
elif artifact_mode == "synthetic":
|
||||
verification_status = "verified-synthetic"
|
||||
else:
|
||||
verification_status = "verified-real"
|
||||
|
||||
bundle = _build_run_bundle(
|
||||
advisory=advisory,
|
||||
profile=profile,
|
||||
run_id=run_id,
|
||||
verification_status=verification_status,
|
||||
verification_mode=verification_mode,
|
||||
artifact_mode=artifact_mode,
|
||||
baseline_refs=[str(run_dir / "logs" / "baseline.json")] if baseline_payload.get("observations") else [],
|
||||
attack_steps=attack_payload.get("steps", []),
|
||||
browser_refs=browser_payload.get("refs", []),
|
||||
container_log_refs=container_logs,
|
||||
request_log_refs=[str(run_dir / "logs" / "attack.json"), str(run_dir / "logs" / "baseline.json")],
|
||||
blocked_reason=blocked_reason,
|
||||
)
|
||||
report_refs = render.render_run(bundle)
|
||||
bundle["report_refs"] = report_refs
|
||||
evidence.write_run_bundle(run_dir, bundle)
|
||||
ensure_dir(RUNS_DIR)
|
||||
write_json(RUNS_DIR / f"{run_id}.json", bundle)
|
||||
render.render_dashboard()
|
||||
print(bundle)
|
||||
result = _execute_case(args.case, run_id=args.run_id, dry_run=args.dry_run, sync_outputs=not args.dry_run)
|
||||
print(result)
|
||||
return 0
|
||||
|
||||
|
||||
@@ -225,7 +357,9 @@ def cmd_run_system(args) -> int:
|
||||
advisories = [item for item in load_json_dir(ADVISORIES_DIR) if item.get("system_id") == args.system]
|
||||
selected = advisories[: args.limit]
|
||||
for advisory in selected:
|
||||
cmd_run_case(argparse.Namespace(case=advisory["canonical_id"], run_id=None, dry_run=args.dry_run))
|
||||
_execute_case(advisory["canonical_id"], run_id=None, dry_run=args.dry_run, sync_outputs=False)
|
||||
if selected and not args.dry_run:
|
||||
_sync_registry_outputs()
|
||||
print({"system": args.system, "count": len(selected)})
|
||||
return 0
|
||||
|
||||
@@ -237,7 +371,9 @@ def cmd_run_batch(args) -> int:
|
||||
task_queue.enqueue_from_registry(only_hotlane=args.only_hotlane, limit=args.limit)
|
||||
items = task_queue.dequeue(limit=args.limit)
|
||||
for item in items:
|
||||
cmd_run_case(argparse.Namespace(case=item["advisory_id"], run_id=None, dry_run=args.dry_run))
|
||||
_execute_case(item["advisory_id"], run_id=None, dry_run=args.dry_run, sync_outputs=False)
|
||||
if items and not args.dry_run:
|
||||
_sync_registry_outputs()
|
||||
print({"processed": len(items)})
|
||||
return 0
|
||||
|
||||
|
||||
@@ -3,15 +3,17 @@ from __future__ import annotations
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict
|
||||
|
||||
from lab.compose import generate_compose
|
||||
from lab.compose import compose_payload, generate_compose
|
||||
from lab.utils import command_available, run
|
||||
|
||||
|
||||
def prepare(profile: Dict[str, Any], run_dir: Path, dry_run: bool = False) -> Dict[str, Any]:
|
||||
compose_path, payload = generate_compose(profile, run_dir)
|
||||
payload = compose_payload(profile)
|
||||
compose_path = run_dir / "compose" / "compose.yaml"
|
||||
result = {
|
||||
"compose_path": str(compose_path),
|
||||
"service_count": len(payload.get("services", {})),
|
||||
"compose_preview": payload,
|
||||
"docker_available": command_available("docker"),
|
||||
"status": "ready",
|
||||
}
|
||||
@@ -19,6 +21,7 @@ def prepare(profile: Dict[str, Any], run_dir: Path, dry_run: bool = False) -> Di
|
||||
result["status"] = "planned"
|
||||
return result
|
||||
|
||||
compose_path, payload = generate_compose(profile, run_dir)
|
||||
if not result["docker_available"]:
|
||||
result["status"] = "blocked-artifact"
|
||||
result["blocked_reason"] = "docker unavailable on this machine"
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from lab.config import CASE_RUNS_DIR, DASHBOARD_DIR, RUNS_DIR
|
||||
from lab.utils import ensure_dir, load_json_dir, read_json, write_json, write_text
|
||||
from lab.config import ADVISORIES_DIR, CASE_RUNS_DIR, DASHBOARD_DIR, RUNS_DIR
|
||||
from lab.utils import ensure_dir, isoformat, load_json_dir, now_utc, write_json, write_text
|
||||
|
||||
|
||||
def mermaid_from_steps(run: Dict[str, Any]) -> str:
|
||||
@@ -24,11 +25,29 @@ def mermaid_from_steps(run: Dict[str, Any]) -> str:
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _relative_ref(run_dir: Path, ref: str) -> str:
|
||||
try:
|
||||
return str(Path(ref).resolve().relative_to(run_dir.resolve()))
|
||||
except ValueError:
|
||||
return ref
|
||||
|
||||
|
||||
def _dashboard_ref(run: Dict[str, Any], ref: str) -> str:
|
||||
try:
|
||||
bundle_dir = Path(run["report_refs"]["bundle_dir"]).resolve()
|
||||
relative = Path(ref).resolve().relative_to(bundle_dir)
|
||||
return f"./runs/{run['run_id']}/{relative.as_posix()}"
|
||||
except Exception:
|
||||
return ref
|
||||
|
||||
|
||||
def render_run(run: Dict[str, Any]) -> Dict[str, str]:
|
||||
run_dir = CASE_RUNS_DIR / run["run_id"]
|
||||
ensure_dir(run_dir / "assets")
|
||||
timeline_path = run_dir / "timeline.mmd"
|
||||
write_text(timeline_path, mermaid_from_steps(run))
|
||||
screenshot_refs = [ref for ref in run.get("browser_refs", []) if ref.endswith((".png", ".jpg", ".jpeg"))]
|
||||
relative_screenshots = [_relative_ref(run_dir, ref) for ref in screenshot_refs]
|
||||
|
||||
md_lines = [
|
||||
f"# Run {run['run_id']}",
|
||||
@@ -44,10 +63,44 @@ def render_run(run: Dict[str, Any]) -> Dict[str, str]:
|
||||
f"- 启动时间: `{run['started_at']}`",
|
||||
f"- 完成时间: `{run['finished_at']}`",
|
||||
f"- 阻塞原因: `{run.get('blocked_reason') or '-'}`",
|
||||
f"- Compose 服务: `{', '.join(run.get('compose_services', [])) or '-'}`",
|
||||
"",
|
||||
"## 运行时间线",
|
||||
"",
|
||||
f"- Mermaid: [{timeline_path.name}]({timeline_path})",
|
||||
"",
|
||||
"| 时间 | 步骤 | 状态 | 说明 |",
|
||||
"|------|------|------|------|",
|
||||
]
|
||||
if run.get("timeline"):
|
||||
for item in run["timeline"]:
|
||||
md_lines.append(
|
||||
f"| `{item.get('at', '')}` | `{item.get('step', '')}` | `{item.get('status', '')}` | {item.get('detail', '') or '-'} |"
|
||||
)
|
||||
else:
|
||||
md_lines.append("| `-` | `-` | `-` | 无时间线 |")
|
||||
md_lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Compose 拓扑",
|
||||
"",
|
||||
f"- Compose 文件: `{', '.join(run.get('compose_refs', [])) or '-'}`",
|
||||
f"- 服务列表: `{', '.join(run.get('compose_services', [])) or '-'}`",
|
||||
"",
|
||||
"## 攻击步骤",
|
||||
"",
|
||||
"| 工具/步骤 | 状态 | 结果 |",
|
||||
"|-----------|------|------|",
|
||||
]
|
||||
)
|
||||
if run.get("attack_steps"):
|
||||
for step in run["attack_steps"]:
|
||||
outcome = step.get("result_path") or step.get("detail") or "-"
|
||||
md_lines.append(f"| `{step.get('tool') or step.get('kind')}` | `{step.get('status', '-')}` | `{outcome}` |")
|
||||
else:
|
||||
md_lines.append("| `-` | `skipped` | `no attack steps` |")
|
||||
md_lines.extend(
|
||||
[
|
||||
"",
|
||||
"## 证据摘要",
|
||||
"",
|
||||
@@ -57,29 +110,45 @@ def render_run(run: Dict[str, Any]) -> Dict[str, str]:
|
||||
f"- 容器日志: `{len(run.get('container_log_refs', []))}`",
|
||||
f"- 请求日志: `{len(run.get('request_log_refs', []))}`",
|
||||
"",
|
||||
"## 最小化验证说明",
|
||||
"",
|
||||
"- 仅限自有资产、本地靶场或已授权实验目标。",
|
||||
"- 默认执行 minimal-proof;不会把破坏性或不可回滚动作作为默认路径。",
|
||||
"",
|
||||
]
|
||||
)
|
||||
if relative_screenshots:
|
||||
md_lines.extend(["## 浏览器截图", ""])
|
||||
for ref in relative_screenshots:
|
||||
md_lines.append(f"")
|
||||
md_lines.append("")
|
||||
if run.get("browser_refs"):
|
||||
md_lines.extend(["## 浏览器证据", ""])
|
||||
for ref in run["browser_refs"]:
|
||||
md_lines.append(f"- {ref}")
|
||||
md_lines.append(f"- `{_relative_ref(run_dir, ref)}`")
|
||||
md_lines.append("")
|
||||
if run.get("container_log_refs"):
|
||||
md_lines.extend(["## 容器日志", ""])
|
||||
for ref in run["container_log_refs"]:
|
||||
md_lines.append(f"- {ref}")
|
||||
md_lines.append(f"- `{_relative_ref(run_dir, ref)}`")
|
||||
md_lines.append("")
|
||||
if run.get("request_log_refs"):
|
||||
md_lines.extend(["## 请求与基线日志", ""])
|
||||
for ref in run["request_log_refs"]:
|
||||
md_lines.append(f"- `{_relative_ref(run_dir, ref)}`")
|
||||
md_lines.append("")
|
||||
md_lines.extend(
|
||||
[
|
||||
"## 最小化验证说明",
|
||||
"",
|
||||
"- 仅限自有资产、本地靶场或已授权实验目标。",
|
||||
"- 默认执行 minimal-proof;不会把破坏性或不可回滚动作作为默认路径。",
|
||||
"- 若浏览器证据缺失,前端类案例不会被标为 `verified-*`。",
|
||||
"",
|
||||
]
|
||||
)
|
||||
report_md = run_dir / "report.md"
|
||||
write_text(report_md, "\n".join(md_lines))
|
||||
|
||||
html_body = [
|
||||
"<!doctype html>",
|
||||
"<html><head><meta charset='utf-8'><title>websafe run report</title>",
|
||||
"<style>body{font-family:ui-monospace,Menlo,monospace;margin:2rem;line-height:1.5;} code,pre{background:#f5f5f5;padding:.2rem .4rem;} .grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:1rem;} .card{border:1px solid #ddd;padding:1rem;border-radius:.5rem;}</style>",
|
||||
"<style>body{font-family:ui-sans-serif,system-ui,sans-serif;margin:2rem;line-height:1.55;background:#f8fafc;color:#0f172a;} code,pre{background:#e2e8f0;padding:.2rem .4rem;border-radius:.3rem;} pre{white-space:pre-wrap;} .grid{display:grid;grid-template-columns:repeat(2,minmax(0,1fr));gap:1rem;} .card{border:1px solid #cbd5e1;padding:1rem;border-radius:.75rem;background:#fff;} table{width:100%;border-collapse:collapse;background:#fff;border:1px solid #cbd5e1;border-radius:.75rem;overflow:hidden;} th,td{padding:.75rem;border-bottom:1px solid #e2e8f0;text-align:left;vertical-align:top;} img{max-width:100%;border:1px solid #cbd5e1;border-radius:.5rem;} .gallery{display:grid;grid-template-columns:repeat(auto-fit,minmax(320px,1fr));gap:1rem;}</style>",
|
||||
"</head><body>",
|
||||
f"<h1>Run {html.escape(run['run_id'])}</h1>",
|
||||
"<div class='grid'>",
|
||||
@@ -90,10 +159,42 @@ def render_run(run: Dict[str, Any]) -> Dict[str, str]:
|
||||
"</div>",
|
||||
"<h2>Mermaid Timeline</h2>",
|
||||
f"<pre>{html.escape(mermaid_from_steps(run))}</pre>",
|
||||
"<h2>Evidence</h2><ul>",
|
||||
"<h2>Timeline</h2>",
|
||||
"<table><thead><tr><th>Time</th><th>Step</th><th>Status</th><th>Detail</th></tr></thead><tbody>",
|
||||
]
|
||||
for ref in run.get("browser_refs", []) + run.get("container_log_refs", []) + run.get("request_log_refs", []):
|
||||
html_body.append(f"<li><code>{html.escape(ref)}</code></li>")
|
||||
if run.get("timeline"):
|
||||
for item in run["timeline"]:
|
||||
html_body.append(
|
||||
"<tr>"
|
||||
f"<td><code>{html.escape(item.get('at', ''))}</code></td>"
|
||||
f"<td><code>{html.escape(item.get('step', ''))}</code></td>"
|
||||
f"<td><code>{html.escape(item.get('status', ''))}</code></td>"
|
||||
f"<td>{html.escape(item.get('detail', '') or '-')}</td>"
|
||||
"</tr>"
|
||||
)
|
||||
html_body.extend(["</tbody></table>", "<h2>Attack Steps</h2>", "<table><thead><tr><th>Tool</th><th>Status</th><th>Output</th></tr></thead><tbody>"])
|
||||
if run.get("attack_steps"):
|
||||
for step in run["attack_steps"]:
|
||||
html_body.append(
|
||||
"<tr>"
|
||||
f"<td><code>{html.escape(step.get('tool') or step.get('kind') or '-')}</code></td>"
|
||||
f"<td><code>{html.escape(step.get('status', '-'))}</code></td>"
|
||||
f"<td><code>{html.escape(step.get('result_path') or '-')}</code></td>"
|
||||
"</tr>"
|
||||
)
|
||||
else:
|
||||
html_body.append("<tr><td><code>-</code></td><td><code>skipped</code></td><td><code>no attack steps</code></td></tr>")
|
||||
html_body.extend(["</tbody></table>"])
|
||||
if relative_screenshots:
|
||||
html_body.extend(["<h2>Browser Screenshots</h2>", "<div class='gallery'>"])
|
||||
for ref in relative_screenshots:
|
||||
html_body.append(
|
||||
f"<figure><img src='{html.escape(ref)}' alt='{html.escape(Path(ref).stem)}'><figcaption><code>{html.escape(ref)}</code></figcaption></figure>"
|
||||
)
|
||||
html_body.append("</div>")
|
||||
html_body.extend(["<h2>Evidence</h2><ul>"])
|
||||
for ref in run.get("compose_refs", []) + run.get("browser_refs", []) + run.get("container_log_refs", []) + run.get("request_log_refs", []):
|
||||
html_body.append(f"<li><code>{html.escape(_relative_ref(run_dir, ref))}</code></li>")
|
||||
html_body.extend(["</ul>", "</body></html>"])
|
||||
report_html = run_dir / "report.html"
|
||||
write_text(report_html, "\n".join(html_body))
|
||||
@@ -102,17 +203,102 @@ def render_run(run: Dict[str, Any]) -> Dict[str, str]:
|
||||
|
||||
def render_dashboard() -> Dict[str, str]:
|
||||
ensure_dir(DASHBOARD_DIR)
|
||||
advisory_records = load_json_dir(ADVISORIES_DIR)
|
||||
runs = load_json_dir(RUNS_DIR)
|
||||
runs_dir = DASHBOARD_DIR / "runs"
|
||||
ensure_dir(runs_dir)
|
||||
for item in runs:
|
||||
bundle_dir = Path(item.get("report_refs", {}).get("bundle_dir", ""))
|
||||
if not bundle_dir.exists():
|
||||
continue
|
||||
symlink_path = runs_dir / item["run_id"]
|
||||
try:
|
||||
if symlink_path.is_symlink() or symlink_path.exists():
|
||||
if symlink_path.is_symlink() and symlink_path.resolve() == bundle_dir.resolve():
|
||||
pass
|
||||
else:
|
||||
symlink_path.unlink()
|
||||
os.symlink(bundle_dir, symlink_path, target_is_directory=True)
|
||||
else:
|
||||
os.symlink(bundle_dir, symlink_path, target_is_directory=True)
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
systems: Dict[str, Dict[str, Any]] = {}
|
||||
for advisory in advisory_records:
|
||||
system = systems.setdefault(
|
||||
advisory["system_id"],
|
||||
{
|
||||
"system_id": advisory["system_id"],
|
||||
"display_name": advisory.get("display_name", advisory["system_id"]),
|
||||
"total": 0,
|
||||
"verified_real": 0,
|
||||
"verified_synthetic": 0,
|
||||
"blocked": 0,
|
||||
"manual": 0,
|
||||
"browser_required": 0,
|
||||
"browser_present": 0,
|
||||
"latest_update": "",
|
||||
},
|
||||
)
|
||||
system["total"] += 1
|
||||
status = advisory.get("verification_status", "triage-manual")
|
||||
if status == "verified-real":
|
||||
system["verified_real"] += 1
|
||||
elif status == "verified-synthetic":
|
||||
system["verified_synthetic"] += 1
|
||||
elif status.startswith("blocked-"):
|
||||
system["blocked"] += 1
|
||||
else:
|
||||
system["manual"] += 1
|
||||
browser = advisory.get("browser_evidence", {})
|
||||
if browser.get("required"):
|
||||
system["browser_required"] += 1
|
||||
if browser.get("present"):
|
||||
system["browser_present"] += 1
|
||||
latest = advisory.get("updated_at") or advisory.get("published_at") or ""
|
||||
if latest > system["latest_update"]:
|
||||
system["latest_update"] = latest
|
||||
|
||||
recent_runs = sorted(runs, key=lambda item: item.get("finished_at") or "", reverse=True)[:100]
|
||||
decorated_runs: List[Dict[str, Any]] = []
|
||||
for item in recent_runs:
|
||||
cloned = dict(item)
|
||||
cloned["dashboard_refs"] = {
|
||||
"report_html": f"./runs/{item['run_id']}/report.html",
|
||||
"report_md": f"./runs/{item['run_id']}/report.md",
|
||||
"timeline": f"./runs/{item['run_id']}/timeline.mmd",
|
||||
"bundle": f"./runs/{item['run_id']}/run.json",
|
||||
}
|
||||
cloned["browser_links"] = [_dashboard_ref(item, ref) for ref in item.get("browser_refs", [])]
|
||||
cloned["container_links"] = [_dashboard_ref(item, ref) for ref in item.get("container_log_refs", [])]
|
||||
cloned["request_links"] = [_dashboard_ref(item, ref) for ref in item.get("request_log_refs", [])]
|
||||
decorated_runs.append(cloned)
|
||||
|
||||
summary = {
|
||||
"generated_at": isoformat(now_utc()),
|
||||
"advisory_count": len(advisory_records),
|
||||
"run_count": len(runs),
|
||||
"statuses": {},
|
||||
"recent_runs": sorted(runs, key=lambda item: item.get("finished_at") or "", reverse=True)[:50],
|
||||
"recent_failures": [],
|
||||
}
|
||||
for item in runs:
|
||||
status = item.get("verification_status", "triage-manual")
|
||||
summary["statuses"][status] = summary["statuses"].get(status, 0) + 1
|
||||
summary["systems"] = sorted(systems.values(), key=lambda item: (-item["total"], item["system_id"]))
|
||||
summary["recent_failures"] = [
|
||||
{
|
||||
"run_id": item["run_id"],
|
||||
"advisory_id": item["advisory_id"],
|
||||
"status": item.get("verification_status"),
|
||||
"blocked_reason": item.get("blocked_reason"),
|
||||
}
|
||||
for item in decorated_runs
|
||||
if item.get("verification_status") in {"triage-manual", "blocked-artifact", "blocked-destructive"}
|
||||
][:20]
|
||||
write_json(DASHBOARD_DIR / "summary.json", summary)
|
||||
write_json(DASHBOARD_DIR / "runs.json", summary["recent_runs"])
|
||||
write_json(DASHBOARD_DIR / "runs.json", decorated_runs)
|
||||
write_json(DASHBOARD_DIR / "systems.json", summary["systems"])
|
||||
|
||||
html_page = """<!doctype html>
|
||||
<html>
|
||||
@@ -124,36 +310,87 @@ def render_dashboard() -> Dict[str, str]:
|
||||
h1, h2 { margin-bottom: .5rem; }
|
||||
.cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 1rem; margin: 1rem 0 2rem; }
|
||||
.card { background: white; border: 1px solid #cbd5e1; border-radius: 14px; padding: 1rem; box-shadow: 0 4px 18px rgba(15,23,42,.06); }
|
||||
table { width: 100%%; border-collapse: collapse; background: white; border-radius: 12px; overflow: hidden; }
|
||||
.filters { display:flex; flex-wrap:wrap; gap:.75rem; margin: 1rem 0; }
|
||||
input, select { padding: .6rem .75rem; border: 1px solid #cbd5e1; border-radius: 10px; background: white; }
|
||||
table { width: 100%%; border-collapse: collapse; background: white; border-radius: 12px; overflow: hidden; margin-bottom: 2rem; }
|
||||
th, td { padding: .75rem; border-bottom: 1px solid #e2e8f0; text-align: left; font-size: .92rem; }
|
||||
code { background: #e2e8f0; padding: .1rem .35rem; border-radius: 6px; }
|
||||
.muted { color: #475569; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>websafe Local Lab Dashboard</h1>
|
||||
<p>LAB ONLY | AUTHORIZED TARGETS ONLY | 本地静态看板</p>
|
||||
<div id="summary" class="cards"></div>
|
||||
<h2>Recent Runs</h2>
|
||||
<h2>System Coverage</h2>
|
||||
<table>
|
||||
<thead><tr><th>Run</th><th>Advisory</th><th>Status</th><th>Mode</th><th>Finished</th><th>Report</th></tr></thead>
|
||||
<thead><tr><th>System</th><th>Total</th><th>Verified Real</th><th>Verified Synthetic</th><th>Blocked</th><th>Manual</th><th>Browser</th><th>Latest</th></tr></thead>
|
||||
<tbody id="systemRows"></tbody>
|
||||
</table>
|
||||
<h2>Recent Runs</h2>
|
||||
<div class="filters">
|
||||
<input id="search" placeholder="Search advisory or run id">
|
||||
<select id="systemFilter"><option value="">All systems</option></select>
|
||||
<select id="statusFilter"><option value="">All statuses</option></select>
|
||||
<select id="familyFilter"><option value="">All profiles</option></select>
|
||||
</div>
|
||||
<table>
|
||||
<thead><tr><th>Run</th><th>System</th><th>Advisory</th><th>Status</th><th>Mode</th><th>Profile</th><th>Finished</th><th>Artifacts</th></tr></thead>
|
||||
<tbody id="rows"></tbody>
|
||||
</table>
|
||||
<script>
|
||||
async function main() {
|
||||
const summary = await fetch('./summary.json').then(r => r.json());
|
||||
const runs = await fetch('./runs.json').then(r => r.json());
|
||||
const [summary, runs, systems] = await Promise.all([
|
||||
fetch('./summary.json').then(r => r.json()),
|
||||
fetch('./runs.json').then(r => r.json()),
|
||||
fetch('./systems.json').then(r => r.json())
|
||||
]);
|
||||
const summaryRoot = document.getElementById('summary');
|
||||
const cards = [{label: 'Run Count', value: summary.run_count}];
|
||||
const cards = [{label: 'Advisories', value: summary.advisory_count}, {label: 'Run Count', value: summary.run_count}];
|
||||
for (const [key, value] of Object.entries(summary.statuses)) {
|
||||
cards.push({label: key, value});
|
||||
}
|
||||
summaryRoot.innerHTML = cards.map(item => `<div class="card"><strong>${item.label}</strong><div style="font-size:2rem;margin-top:.5rem;">${item.value}</div></div>`).join('');
|
||||
|
||||
const systemRows = document.getElementById('systemRows');
|
||||
systemRows.innerHTML = systems.map(item => `<tr><td><code>${item.system_id}</code></td><td>${item.total}</td><td>${item.verified_real}</td><td>${item.verified_synthetic}</td><td>${item.blocked}</td><td>${item.manual}</td><td>${item.browser_present}/${item.browser_required}</td><td>${item.latest_update || ''}</td></tr>`).join('');
|
||||
|
||||
const systemFilter = document.getElementById('systemFilter');
|
||||
const statusFilter = document.getElementById('statusFilter');
|
||||
const familyFilter = document.getElementById('familyFilter');
|
||||
const search = document.getElementById('search');
|
||||
const distinct = (values) => Array.from(new Set(values.filter(Boolean))).sort();
|
||||
systemFilter.innerHTML += distinct(runs.map(item => item.system_id)).map(value => `<option value="${value}">${value}</option>`).join('');
|
||||
statusFilter.innerHTML += distinct(runs.map(item => item.verification_status)).map(value => `<option value="${value}">${value}</option>`).join('');
|
||||
familyFilter.innerHTML += distinct(runs.map(item => item.repro_profile_id)).map(value => `<option value="${value}">${value}</option>`).join('');
|
||||
|
||||
const rows = document.getElementById('rows');
|
||||
rows.innerHTML = runs.map(item => {
|
||||
const report = item.report_refs && item.report_refs.report_html ? item.report_refs.report_html : '';
|
||||
return `<tr><td><code>${item.run_id}</code></td><td><code>${item.advisory_id}</code></td><td>${item.verification_status}</td><td>${item.verification_mode}</td><td>${item.finished_at || ''}</td><td>${report ? `<a href="../../../../${report.replace('/Users/x/websafe/', '')}">open</a>` : '-'}</td></tr>`;
|
||||
function renderRows() {
|
||||
const query = search.value.trim().toLowerCase();
|
||||
const filtered = runs.filter(item => {
|
||||
if (systemFilter.value && item.system_id !== systemFilter.value) return false;
|
||||
if (statusFilter.value && item.verification_status !== statusFilter.value) return false;
|
||||
if (familyFilter.value && item.repro_profile_id !== familyFilter.value) return false;
|
||||
if (query) {
|
||||
const haystack = `${item.run_id} ${item.advisory_id} ${item.system_id} ${item.repro_profile_id}`.toLowerCase();
|
||||
if (!haystack.includes(query)) return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
rows.innerHTML = filtered.map(item => {
|
||||
const links = [];
|
||||
if (item.dashboard_refs && item.dashboard_refs.report_html) links.push(`<a href="${item.dashboard_refs.report_html}">report</a>`);
|
||||
if (item.dashboard_refs && item.dashboard_refs.timeline) links.push(`<a href="${item.dashboard_refs.timeline}">timeline</a>`);
|
||||
if (item.dashboard_refs && item.dashboard_refs.bundle) links.push(`<a href="${item.dashboard_refs.bundle}">bundle</a>`);
|
||||
if (item.browser_links && item.browser_links.length) links.push(`<a href="${item.browser_links[0]}">browser</a>`);
|
||||
if (item.container_links && item.container_links.length) links.push(`<a href="${item.container_links[0]}">logs</a>`);
|
||||
const reason = item.blocked_reason ? `<div class="muted">${item.blocked_reason}</div>` : '';
|
||||
return `<tr><td><code>${item.run_id}</code>${reason}</td><td><code>${item.system_id}</code></td><td><code>${item.advisory_id}</code></td><td>${item.verification_status}</td><td>${item.verification_mode}</td><td><code>${item.repro_profile_id}</code></td><td>${item.finished_at || ''}</td><td>${links.join(' | ') || '-'}</td></tr>`;
|
||||
}).join('');
|
||||
}
|
||||
[systemFilter, statusFilter, familyFilter, search].forEach(node => node.addEventListener('input', renderRows));
|
||||
renderRows();
|
||||
}
|
||||
main();
|
||||
</script>
|
||||
</body>
|
||||
|
||||
@@ -1,11 +1,15 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from lab.config import ADVISORIES_DIR, QUEUE_PATH
|
||||
from lab.utils import load_json_dir, read_json, write_json
|
||||
|
||||
|
||||
UTC = timezone.utc
|
||||
|
||||
|
||||
def load_queue() -> Dict[str, Any]:
|
||||
return read_json(QUEUE_PATH, default={"items": []}) or {"items": []}
|
||||
|
||||
@@ -28,15 +32,64 @@ def enqueue_items(items: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
return {"queued": len(queue["items"]), "added": added}
|
||||
|
||||
|
||||
def _parse_iso(value: str | None) -> datetime:
|
||||
if not value:
|
||||
return datetime(1970, 1, 1, tzinfo=UTC)
|
||||
try:
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone(UTC)
|
||||
except ValueError:
|
||||
return datetime(1970, 1, 1, tzinfo=UTC)
|
||||
|
||||
|
||||
def _priority_tuple(advisory: Dict[str, Any], only_hotlane: bool) -> tuple[int, float]:
|
||||
score = 0
|
||||
verification_status = advisory.get("verification_status", "triage-manual")
|
||||
if verification_status == "triage-manual":
|
||||
score += 500
|
||||
elif verification_status.startswith("blocked-"):
|
||||
score += 450
|
||||
elif verification_status == "verified-synthetic":
|
||||
score += 300
|
||||
else:
|
||||
score += 150
|
||||
|
||||
last_verified = _parse_iso(advisory.get("last_verified_at"))
|
||||
latest_upstream = max(_parse_iso(advisory.get("updated_at")), _parse_iso(advisory.get("published_at")))
|
||||
if advisory.get("last_verified_at") is None:
|
||||
score += 350
|
||||
elif latest_upstream > last_verified:
|
||||
score += 250
|
||||
|
||||
exploit_status = advisory.get("exploit_status")
|
||||
if exploit_status in {"known_exploited", "active_exploitation", "in_the_wild"}:
|
||||
score += 1000
|
||||
severity = advisory.get("severity")
|
||||
if severity == "critical":
|
||||
score += 250
|
||||
score += int((advisory.get("cvss_score") or 0) * 10)
|
||||
|
||||
if only_hotlane:
|
||||
score += 100
|
||||
|
||||
return score, latest_upstream.timestamp()
|
||||
|
||||
|
||||
def enqueue_from_registry(only_hotlane: bool = False, limit: int = 50) -> Dict[str, Any]:
|
||||
advisories = load_json_dir(ADVISORIES_DIR)
|
||||
advisories = sorted(advisories, key=lambda item: _priority_tuple(item, only_hotlane), reverse=True)
|
||||
items = []
|
||||
for advisory in advisories:
|
||||
if only_hotlane:
|
||||
hot = advisory.get("exploit_status") in {"known_exploited", "active_exploitation", "in_the_wild"}
|
||||
if not hot and not (advisory.get("cvss_score") or 0) >= 8.8 and advisory.get("severity") != "critical":
|
||||
continue
|
||||
items.append({"advisory_id": advisory["canonical_id"], "system_id": advisory["system_id"], "priority": "hotlane" if only_hotlane else "default"})
|
||||
items.append(
|
||||
{
|
||||
"advisory_id": advisory["canonical_id"],
|
||||
"system_id": advisory["system_id"],
|
||||
"priority": "hotlane" if only_hotlane else "default",
|
||||
}
|
||||
)
|
||||
return enqueue_items(items[:limit])
|
||||
|
||||
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from lab.compose import compose_payload
|
||||
from lab.config import ENV_CATALOG_DIR, REPRO_MAP_PATH, REPRO_PROFILES_DIR
|
||||
from lab.utils import read_yaml
|
||||
from lab.utils import command_available, read_yaml, run, write_yaml
|
||||
|
||||
|
||||
def validate_assets() -> List[str]:
|
||||
@@ -32,4 +34,21 @@ def validate_assets() -> List[str]:
|
||||
]:
|
||||
if field not in content:
|
||||
errors.append(f"repro profile missing {field}: {path}")
|
||||
docker_available = command_available("docker")
|
||||
profile_roots = sorted((ENV_CATALOG_DIR.parent.parent / "profiles").rglob("*.yaml"))
|
||||
for path in profile_roots:
|
||||
content = read_yaml(path, default=None)
|
||||
if not isinstance(content, dict):
|
||||
errors.append(f"invalid environment profile yaml: {path}")
|
||||
continue
|
||||
for field in ["profile_id", "system_id", "services", "cleanup_policy"]:
|
||||
if field not in content:
|
||||
errors.append(f"environment profile missing {field}: {path}")
|
||||
if docker_available:
|
||||
with TemporaryDirectory() as temp_dir:
|
||||
compose_path = Path(temp_dir) / "compose.yaml"
|
||||
write_yaml(compose_path, compose_payload(content))
|
||||
result = run(["docker", "compose", "-f", str(compose_path), "config"], cwd=compose_path.parent)
|
||||
if result.returncode != 0:
|
||||
errors.append(f"docker compose config failed for {path}: {result.stderr.strip() or result.stdout.strip()}")
|
||||
return errors
|
||||
|
||||
在新工单中引用
屏蔽一个用户