Pierre-Olivier Mercier
c2cc88e1df
Downloads and caches the Disconnect.me services.json feed (24h TTL), matching domains against the Advertising, Analytics, Social, Content, and Disconnect categories. Severity is warn (privacy classification, not malware). Reuses the shared feedCache infrastructure.
134 lines
5.5 KiB
Markdown
134 lines
5.5 KiB
Markdown
# checker-blacklist
|
|
|
|
happyDomain checker that flags whether a domain is currently listed on
|
|
widely-used reputation systems.
|
|
|
|
## Sources
|
|
|
|
| Source | Type | API key needed | Configurable |
|
|
|-----------------------|-----------------|----------------|--------------|
|
|
| Spamhaus DBL | DNS-based DBL | no | admin (default on) |
|
|
| SURBL multi | DNS-based DBL | no | admin (default on) |
|
|
| URIBL multi | DNS-based DBL | no | admin (default on) |
|
|
| Extra DNSBL zones | DNS-based DBL | no | admin |
|
|
| Google Safe Browsing | HTTPS lookup | yes (admin) | admin |
|
|
| OpenPhish public feed | downloaded list | no | user (default on) |
|
|
| PhishTank | downloaded list | no | user (default on) |
|
|
| abuse.ch URLhaus | HTTPS lookup | free Auth-Key (admin) | user (default on) |
|
|
| abuse.ch ThreatFox | HTTPS lookup | free Auth-Key (admin) | user (default on) |
|
|
| abuse.ch MalwareBazaar| HTTPS lookup | free Auth-Key (admin) | user (default on) |
|
|
| Botvrij.eu | downloaded list | no | user (default on) |
|
|
| Disconnect.me | downloaded list | no | user (default on) |
|
|
| VirusTotal v3 | HTTPS lookup | yes (admin) | admin |
|
|
|
|
### Obtaining API keys
|
|
|
|
**Google Safe Browsing** (option: `google_safe_browsing_api_key`)
|
|
1. Go to the [Google Cloud Console](https://console.cloud.google.com/) and create or select a project.
|
|
2. Enable the *Safe Browsing API* under *APIs & Services → Library*.
|
|
3. Create an API key under *APIs & Services → Credentials*.
|
|
4. The free tier allows up to 10 000 queries/day with no billing required.
|
|
|
|
**abuse.ch** (option: `urlhaus_auth_key` / `threatfox_auth_key` / `malwarebazaar_auth_key`)
|
|
1. Register a free account at [abuse.ch](https://abuse.ch/).
|
|
2. After login, retrieve your Auth-Key from your account profile page.
|
|
3. The same account and key works for URLhaus, ThreatFox, and MalwareBazaar — set it in each source option independently.
|
|
4. Free, no rate-limit tiers documented; the APIs are community-funded.
|
|
|
|
**VirusTotal** (option: `virustotal_api_key`)
|
|
1. Create a free account at [virustotal.com](https://www.virustotal.com/).
|
|
2. Go to your profile and copy the API key.
|
|
3. Free tier: 4 requests/minute, 500 requests/day. No billing required.
|
|
4. The public API key is sufficient; premium keys unlock higher quotas.
|
|
|
|
DNS-based blocklists are queried in parallel. The OpenPhish feed is
|
|
downloaded once per hour by the provider and cached in memory.
|
|
|
|
## Common failure scenarios surfaced in the HTML report
|
|
|
|
The report opens with a diagnosis-first "Action required" section that
|
|
lists the most common, high-impact problems with a one-shot remediation:
|
|
|
|
1. **Listed on Spamhaus DBL / SURBL / URIBL**: direct lookup link and
|
|
removal procedure URL per operator.
|
|
2. **Flagged by Google Safe Browsing**: link to Google Search Console's
|
|
security-issues review request.
|
|
3. **Listed in the OpenPhish feed**: instructions to treat the host as
|
|
compromised (audit recently-added files, rotate credentials), plus a
|
|
link to OpenPhish feedback.
|
|
4. **Listed in URLhaus (active malware distribution)**: direct link to
|
|
the abuse.ch reference page and per-URL takedown notification flow.
|
|
5. **VirusTotal multi-vendor flag**: Critical when at least one vendor
|
|
reports `malicious`, Warning when only `suspicious`. Lists the
|
|
flagging engines and links to the VT GUI page for re-scan / vendor
|
|
contact.
|
|
6. **DNSBL query refused / API quota exhausted**: most public resolvers
|
|
are blocked by DBL/URIBL operators; surfaced as a warning so it does
|
|
not pollute the OK status.
|
|
|
|
A per-source detail table follows for full context (return codes, TXT
|
|
records, threat types, sample phishing URLs).
|
|
|
|
## Adding a new source
|
|
|
|
Every reputation backend implements the `Source` interface in its own
|
|
file and registers itself from `init()`. Skeleton:
|
|
|
|
```go
|
|
package checker
|
|
|
|
import (
|
|
"context"
|
|
sdk "git.happydns.org/checker-sdk-go/checker"
|
|
)
|
|
|
|
func init() { Register(&mySource{}) }
|
|
|
|
type mySource struct{}
|
|
|
|
func (*mySource) ID() string { return "mybl" }
|
|
func (*mySource) Name() string { return "My Blocklist" }
|
|
|
|
func (*mySource) Options() SourceOptions {
|
|
return SourceOptions{
|
|
Admin: []sdk.CheckerOptionField{ /* … */ },
|
|
}
|
|
}
|
|
|
|
func (*mySource) Query(ctx context.Context, domain, registered string, opts sdk.CheckerOptions) []SourceResult {
|
|
res := SourceResult{SourceID: "mybl", SourceName: "My Blocklist", Enabled: true}
|
|
// …populate Listed / Severity / Reasons / Evidence / Reference / Error
|
|
return []SourceResult{res}
|
|
}
|
|
|
|
func (*mySource) Diagnose(res SourceResult) Diagnosis {
|
|
return Diagnosis{Severity: SeverityCrit, Title: "Listed", Detail: "…"}
|
|
}
|
|
```
|
|
|
|
That's it: rules, the report, metrics, the standalone `/check` form
|
|
and the definition pick the new source up automatically. Sources that
|
|
need richer rendering (a per-vendor table, etc.) additionally
|
|
implement `RenderDetail(SourceResult) (template.HTML, error)`.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
make # standalone binary (HTTP server + /check form)
|
|
make plugin # checker-blacklist.so for happyDomain dynamic load
|
|
make docker # container image
|
|
```
|
|
|
|
## Running standalone
|
|
|
|
```bash
|
|
./checker-blacklist -listen :8080
|
|
# then GET /check, /definition, /health, …
|
|
```
|
|
|
|
The standalone binary embeds an `interactive` form on `GET /check` so a
|
|
human can paste a domain and run the full pipeline without happyDomain.
|
|
|
|
## License
|
|
|
|
MIT (checker code); Apache 2.0 SDK dependency.
|