hao/websafe-kb
1
0
派生 0
代码 工单 合并请求 1 工作流 软件包 项目 发布 百科 活动

更新: 4 个文件 - 2026-03-18 17:23:40

浏览代码
这个提交包含在：
hao
2026-03-18 17:23:40 -07:00
父节点 301d15e91e
当前提交 6dff954778
共修改 4 个文件，包含 445 行新增和 134 行删除
下载 Patch 文件 下载 Diff 文件 展开所有文件 折叠所有文件

353
08-threat-intel/generated/source-health.json
查看文件

文件差异内容过多而无法显示 加载差异

169
08-threat-intel/source-map.yaml
查看文件

@@ -78,6 +78,9 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 50
status: retired
retired_reason: OSV Drupal + Drupal official RSS now cover machine-readable collection with lower cold-start latency than NVD public search.
replacement_sources: [Drupal Security Advisories RSS, OSV Drupal]
ecosystem_sources:
- name: Drupal Security Advisories Site
kind: html-links
@@ -98,8 +101,13 @@ systems:
retired_reason: Unauthenticated GHSA API requests are rate-limited in daily monitoring; RSS and NVD remain active replacements.
replacement_sources: [Drupal Security Advisories RSS, NVD Drupal]
research_sources: []
ecosystem_sources:
- name: OSV Drupal
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
package_names:
- ecosystem: composer
- ecosystem: Packagist
name: drupal/core
cpe_keys: ["drupal:drupal"]
ghsa_keywords: [drupal, drupal core]
@@ -129,9 +137,18 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 50
ecosystem_sources: []
status: retired
retired_reason: OSV Joomla CMS replaces NVD for machine-readable collection without public NVD throttling.
replacement_sources: [Joomla Security Centre, OSV Joomla]
ecosystem_sources:
- name: OSV Joomla
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names: []
package_names:
- ecosystem: Packagist
name: joomla/joomla-cms
cpe_keys: ["joomla:joomla!"]
ghsa_keywords: [joomla]
kev_keywords: [joomla]
@@ -160,7 +177,14 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: OSV Ghost replaces NVD for machine-readable collection and keeps npm package alignment.
replacement_sources: [Ghost GitHub Advisories, OSV Ghost]
ecosystem_sources:
- name: OSV Ghost
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names:
- ecosystem: npm
@@ -267,9 +291,18 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: MediaWiki announce RSS plus OSV MediaWiki now replace NVD for lower-latency machine-readable collection.
replacement_sources: [MediaWiki Announce RSS, OSV MediaWiki]
ecosystem_sources:
- name: OSV MediaWiki
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names: []
package_names:
- ecosystem: Packagist
name: mediawiki/core
cpe_keys: ["mediawiki:mediawiki"]
ghsa_keywords: [mediawiki]
kev_keywords: [mediawiki]
@@ -306,9 +339,18 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: OSV Moodle replaces NVD for machine-readable collection while official Moodle sources remain for cross-checking.
replacement_sources: [OSV Moodle]
ecosystem_sources:
- name: OSV Moodle
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names: []
package_names:
- ecosystem: Packagist
name: moodle/moodle
cpe_keys: ["moodle:moodle"]
ghsa_keywords: [moodle]
kev_keywords: [moodle]
@@ -504,10 +546,17 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: OSV OpenMage replaces NVD for machine-readable composer-aligned collection.
replacement_sources: [OpenMage GitHub Advisories, OSV OpenMage]
ecosystem_sources:
- name: OSV OpenMage
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names:
- ecosystem: composer
- ecosystem: Packagist
name: openmage/magento-lts
cpe_keys: []
ghsa_keywords: [openmage, mage-os]
@@ -543,7 +592,14 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV WooCommerce replaces NVD for machine-readable collection while official and ecosystem advisory pages remain active.
replacement_sources: [Woo Developer Advisories, GitHub WooCommerce Advisories, OSV WooCommerce]
ecosystem_sources:
- name: OSV WooCommerce
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
- name: Patchstack Database
kind: html-links
url: https://patchstack.com/database/
@@ -562,7 +618,7 @@ systems:
package_names:
- ecosystem: npm
name: "@woocommerce/blocks"
- ecosystem: composer
- ecosystem: Packagist
name: woocommerce/woocommerce
cpe_keys: []
ghsa_keywords: [woocommerce]
@@ -599,7 +655,14 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV PrestaShop replaces NVD for machine-readable collection while official and ecosystem advisories remain active.
replacement_sources: [PrestaShop Security Page, GitHub PrestaShop Advisories, OSV PrestaShop]
ecosystem_sources:
- name: OSV PrestaShop
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
- name: Friends Of Presta Security
kind: html-links
url: https://security.friendsofpresta.org/
@@ -609,7 +672,7 @@ systems:
max_items: 50
research_sources: []
package_names:
- ecosystem: composer
- ecosystem: Packagist
name: prestashop/prestashop
cpe_keys: ["prestashop:prestashop"]
ghsa_keywords: [prestashop]
@@ -639,10 +702,17 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: OSV Shopware replaces NVD for machine-readable collection with lower cold-start overhead.
replacement_sources: [Shopware Security Advisories, OSV Shopware]
ecosystem_sources:
- name: OSV Shopware
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names:
- ecosystem: composer
- ecosystem: Packagist
name: shopware/platform
cpe_keys: []
ghsa_keywords: [shopware]
@@ -671,10 +741,17 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 50
ecosystem_sources: []
status: retired
retired_reason: OSV OpenCart replaces NVD for machine-readable collection while official release source remains active.
replacement_sources: [OpenCart Releases, OSV OpenCart]
ecosystem_sources:
- name: OSV OpenCart
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names:
- ecosystem: composer
- ecosystem: Packagist
name: opencart/opencart
cpe_keys: ["opencart:opencart"]
ghsa_keywords: [opencart]
@@ -703,10 +780,17 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: OSV Saleor replaces NVD for machine-readable collection and aligns with the published PyPI package.
replacement_sources: [GitHub Saleor Advisories, OSV Saleor]
ecosystem_sources:
- name: OSV Saleor
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names:
- ecosystem: pypi
- ecosystem: PyPI
name: saleor
cpe_keys: []
ghsa_keywords: [saleor]
@@ -1069,6 +1153,9 @@ systems:
confidence: ecosystem-authority
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV Express replaces NVD public search for lower-latency machine-readable collection.
replacement_sources: [OSV Express]
research_sources: []
package_names:
- ecosystem: npm
@@ -1107,6 +1194,9 @@ systems:
confidence: ecosystem-authority
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV NestJS replaces NVD public search for lower-latency machine-readable collection.
replacement_sources: [OSV NestJS]
research_sources: []
package_names:
- ecosystem: npm
@@ -1271,6 +1361,9 @@ systems:
confidence: ecosystem-authority
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV Undici replaces NVD public search for lower-latency machine-readable collection.
replacement_sources: [OSV Undici]
research_sources: []
package_names:
- ecosystem: npm
@@ -1309,6 +1402,9 @@ systems:
confidence: ecosystem-authority
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV webpack replaces NVD public search for lower-latency machine-readable collection.
replacement_sources: [OSV webpack]
research_sources: []
package_names:
- ecosystem: npm
@@ -1347,6 +1443,9 @@ systems:
confidence: ecosystem-authority
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV esbuild replaces NVD public search for lower-latency machine-readable collection.
replacement_sources: [OSV esbuild]
research_sources: []
package_names:
- ecosystem: npm
@@ -1687,6 +1786,9 @@ systems:
confidence: ecosystem-authority
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: OSV Rails replaces NVD public search for lower-latency machine-readable collection.
replacement_sources: [OSV Rails]
research_sources: []
package_names:
- ecosystem: RubyGems
@@ -1995,7 +2097,14 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: OSV phpMyAdmin replaces NVD for machine-readable collection while the official security page remains active.
replacement_sources: [phpMyAdmin Security Page, OSV phpMyAdmin]
ecosystem_sources:
- name: OSV phpMyAdmin
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names:
- ecosystem: Packagist
@@ -2211,6 +2320,9 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
status: retired
retired_reason: Mattermost official JSON feed plus OSV Mattermost replace NVD for lower-latency machine-readable collection.
replacement_sources: [Mattermost Security Updates JSON, OSV Mattermost]
- name: Mattermost Security Updates JSON
kind: json-feed
url: https://securityupdates.mattermost.com/security_updates.json
@@ -2219,9 +2331,15 @@ systems:
max_items: 600
request_policy:
accept: application/json
ecosystem_sources: []
ecosystem_sources:
- name: OSV Mattermost
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names: []
package_names:
- ecosystem: Go
name: github.com/mattermost/mattermost-server
cpe_keys: ["mattermost:mattermost"]
ghsa_keywords: [mattermost]
kev_keywords: [mattermost]
@@ -2249,7 +2367,14 @@ systems:
confidence: official
advisory_mode: core
results_per_page: 40
ecosystem_sources: []
status: retired
retired_reason: Official Redmine advisories page remains active and NVD public search is retired to reduce cold-start latency.
replacement_sources: [Redmine Security Advisories]
ecosystem_sources:
- name: OSV Redmine
kind: osv-batch
confidence: ecosystem-authority
advisory_mode: core
research_sources: []
package_names:
- ecosystem: RubyGems

32
scripts/intel/monitoring.py
查看文件

@@ -206,6 +206,37 @@ def build_source_health_snapshot(
all_green = not normalized_failures
previous = previous or {}
last_fully_green_run = generated_at if all_green else previous.get("last_fully_green_run")
slow_sources = []
telemetry_rows = []
for probe in probes:
if probe.get("elapsed_seconds") is None:
continue
telemetry_rows.append(
{
"system_id": probe["system_id"],
"source_name": probe["source_name"],
"source_kind": probe.get("source_kind"),
"elapsed_seconds": probe.get("elapsed_seconds"),
"status": "ok",
}
)
for failure in normalized_failures:
if failure.get("elapsed_seconds") is None:
continue
telemetry_rows.append(
{
"system_id": failure["system_id"],
"source_name": failure["source_name"],
"source_kind": failure.get("source_kind"),
"elapsed_seconds": failure.get("elapsed_seconds"),
"status": failure.get("category") or "failure",
}
)
slow_sources = sorted(
telemetry_rows,
key=lambda item: float(item.get("elapsed_seconds") or 0),
reverse=True,
)[:10]
return {
"generated_at": generated_at,
"active_source_count": active_source_total,
@@ -216,6 +247,7 @@ def build_source_health_snapshot(
"retries_performed": retries_performed,
"probes": sorted(probes, key=lambda item: (item["system_id"], item["source_name"])),
"failures": normalized_failures,
"slow_sources": slow_sources,
"systems": sorted(systems.values(), key=lambda item: item["system_id"]),
}

25
scripts/intel/sources/runner.py
查看文件

@@ -4,6 +4,7 @@ import os
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
from time import perf_counter
from typing import Any, Dict, List, Optional, Tuple
import requests
@@ -56,13 +57,19 @@ def failure_summary(failure: Dict[str, Any]) -> str:
return failure.get("summary") or f"{failure.get('system_id')}::{failure.get('source_name')}::{failure.get('category')}::{failure.get('exception')}"
def build_failure(system: Dict[str, Any], source: Dict[str, Any], exc: Exception) -> Dict[str, Any]:
def build_failure(
system: Dict[str, Any],
source: Dict[str, Any],
exc: Exception,
*,
elapsed_seconds: float | None = None,
) -> Dict[str, Any]:
response = getattr(exc, "response", None)
status_code = getattr(response, "status_code", None)
category = _failure_category(exc)
message = str(exc).strip() or exc.__class__.__name__
summary = f"{system['system_id']}::{source['name']}::{category}::{message}"
return {
failure = {
"system_id": system["system_id"],
"display_name": system["display_name"],
"source_name": source["name"],
@@ -75,6 +82,9 @@ def build_failure(system: Dict[str, Any], source: Dict[str, Any], exc: Exception
"url": source.get("url") or "",
"summary": summary,
}
if elapsed_seconds is not None:
failure["elapsed_seconds"] = round(elapsed_seconds, 3)
return failure
def _collect_jobs(
@@ -113,6 +123,7 @@ def _collect_source_candidates(
since_dt: Optional[datetime],
include_undated: bool,
) -> Tuple[List[Candidate], Optional[Dict[str, Any]]]:
started = perf_counter()
handler = HANDLERS.get(source["kind"])
if handler is None:
return (
@@ -136,7 +147,7 @@ def _collect_source_candidates(
filtered = [item for item in items if _passes_since(item, since_dt, include_undated)]
return filtered, None
except Exception as exc:
return [], build_failure(system, source, exc)
return [], build_failure(system, source, exc, elapsed_seconds=perf_counter() - started)
def probe_source(system: Dict[str, Any], source: Dict[str, Any]) -> Dict[str, Any]:
@@ -279,9 +290,10 @@ def probe_sources(
probes: List[Dict[str, Any]] = []
failures: List[Dict[str, Any]] = []
with ThreadPoolExecutor(max_workers=_max_workers(len(jobs))) as executor:
future_map = {executor.submit(probe_source, system, source): (system, source) for system, source in jobs}
future_map = {executor.submit(probe_source, system, source): (system, source, perf_counter()) for system, source in jobs}
for future in as_completed(future_map):
system, source = future_map[future]
system, source, started = future_map[future]
elapsed = perf_counter() - started
try:
result = future.result()
probes.append(
@@ -289,11 +301,12 @@ def probe_sources(
"system_id": system["system_id"],
"source_name": source["name"],
"source_kind": source["kind"],
"elapsed_seconds": round(elapsed, 3),
**result,
}
)
except Exception as exc:
failures.append(build_failure(system, source, exc))
failures.append(build_failure(system, source, exc, elapsed_seconds=elapsed))
probes.sort(key=lambda item: (item["system_id"], item["source_name"]))
failures.sort(key=lambda item: (item.get("system_id", ""), item.get("source_name", ""), item.get("category", "")))
return probes, failures