Pierre-Olivier Mercier
219d9353c3
Detects domains blocked by Quad9's threat intelligence by comparing the secure resolver (9.9.9.9) against the unsecured peer (9.9.9.10). No API key required; enabled by default via the enable_quad9 user option.
158 lines
6.9 KiB
Markdown
158 lines
6.9 KiB
Markdown
# checker-blacklist
|
|
|
|
happyDomain checker that flags whether a domain is currently listed on
|
|
widely-used reputation systems.
|
|
|
|
## Sources
|
|
|
|
| Source | Type | API key needed | Configurable |
|
|
|-----------------------|-----------------|----------------|--------------|
|
|
| Spamhaus DBL | DNS-based DBL | no | admin (default on) |
|
|
| SURBL multi | DNS-based DBL | no | admin (default on) |
|
|
| URIBL multi | DNS-based DBL | no | admin (default on) |
|
|
| NordSpam DBL | DNS-based DBL | no | admin (default on) |
|
|
| SpamEatingMonkey Fresh| DNS-based DBL | no | admin (default on) |
|
|
| Tiopan DBL | DNS-based DBL | no | admin (default on) |
|
|
| SORBS RHSBL | DNS-based DBL | no | admin (default on) |
|
|
| Extra DNSBL zones | DNS-based DBL | no | admin |
|
|
| Google Safe Browsing | HTTPS lookup | yes (admin) | admin |
|
|
| OpenPhish public feed | downloaded list | no | user (default on) |
|
|
| PhishTank | downloaded list | no | user (default on) |
|
|
| abuse.ch URLhaus | HTTPS lookup | free Auth-Key (admin) | user (default on) |
|
|
| abuse.ch ThreatFox | HTTPS lookup | free Auth-Key (admin) | user (default on) |
|
|
| abuse.ch MalwareBazaar| HTTPS lookup | free Auth-Key (admin) | user (default on) |
|
|
| Botvrij.eu | downloaded list | no | user (default on) |
|
|
| Disconnect.me | downloaded list | no | user (default on) |
|
|
| OISD | downloaded list | no | user (default on) |
|
|
| VirusTotal v3 | HTTPS lookup | yes (admin) | admin |
|
|
| AlienVault OTX | HTTPS lookup | free (admin) | admin |
|
|
| Pulsedive | HTTPS lookup | free (admin) | admin |
|
|
| Criminal IP | HTTPS lookup | yes (admin) | admin |
|
|
| Quad9 secure DNS | DNS comparison | no | user (default on) |
|
|
|
|
### Obtaining API keys
|
|
|
|
**Google Safe Browsing** (option: `google_safe_browsing_api_key`)
|
|
1. Go to the [Google Cloud Console](https://console.cloud.google.com/) and create or select a project.
|
|
2. Enable the *Safe Browsing API* under *APIs & Services → Library*.
|
|
3. Create an API key under *APIs & Services → Credentials*.
|
|
4. The free tier allows up to 10 000 queries/day with no billing required.
|
|
|
|
**abuse.ch** (option: `urlhaus_auth_key` / `threatfox_auth_key` / `malwarebazaar_auth_key`)
|
|
1. Register a free account at [abuse.ch](https://abuse.ch/).
|
|
2. After login, retrieve your Auth-Key from your account profile page.
|
|
3. The same account and key works for URLhaus, ThreatFox, and MalwareBazaar — set it in each source option independently.
|
|
4. Free, no rate-limit tiers documented; the APIs are community-funded.
|
|
|
|
**VirusTotal** (option: `virustotal_api_key`)
|
|
1. Create a free account at [virustotal.com](https://www.virustotal.com/).
|
|
2. Go to your profile and copy the API key.
|
|
3. Free tier: 4 requests/minute, 500 requests/day. No billing required.
|
|
4. The public API key is sufficient; premium keys unlock higher quotas.
|
|
|
|
**AlienVault OTX** (option: `otx_api_key`)
|
|
1. Register a free account at [otx.alienvault.com](https://otx.alienvault.com/).
|
|
2. Go to *Settings → API Integration* to find your personal OTX key.
|
|
3. Free, no documented rate limits for the indicator lookup API.
|
|
|
|
**Pulsedive** (option: `pulsedive_api_key`)
|
|
1. Register a free account at [pulsedive.com](https://pulsedive.com/).
|
|
2. Go to your profile and copy the API key shown under *API*.
|
|
3. Free tier available; higher quotas with a paid plan.
|
|
|
|
**Criminal IP** (option: `criminal_ip_api_key`)
|
|
1. Register a free account at [criminalip.io](https://www.criminalip.io/).
|
|
2. Go to *My Information → API Key* to find your key.
|
|
3. Free tier: 100 requests/day. Paid plans unlock higher quotas.
|
|
|
|
DNS-based blocklists are queried in parallel. The OpenPhish feed is
|
|
downloaded once per hour by the provider and cached in memory.
|
|
|
|
## Common failure scenarios surfaced in the HTML report
|
|
|
|
The report opens with a diagnosis-first "Action required" section that
|
|
lists the most common, high-impact problems with a one-shot remediation:
|
|
|
|
1. **Listed on Spamhaus DBL / SURBL / URIBL / NordSpam / SpamEatingMonkey / Tiopan / SORBS**: direct lookup link and
|
|
removal procedure URL per operator.
|
|
2. **Flagged by Google Safe Browsing**: link to Google Search Console's
|
|
security-issues review request.
|
|
3. **Listed in the OpenPhish feed**: instructions to treat the host as
|
|
compromised (audit recently-added files, rotate credentials), plus a
|
|
link to OpenPhish feedback.
|
|
4. **Listed in URLhaus (active malware distribution)**: direct link to
|
|
the abuse.ch reference page and per-URL takedown notification flow.
|
|
5. **VirusTotal multi-vendor flag**: Critical when at least one vendor
|
|
reports `malicious`, Warning when only `suspicious`. Lists the
|
|
flagging engines and links to the VT GUI page for re-scan / vendor
|
|
contact.
|
|
6. **DNSBL query refused / API quota exhausted**: most public resolvers
|
|
are blocked by DBL/URIBL operators; surfaced as a warning so it does
|
|
not pollute the OK status.
|
|
|
|
A per-source detail table follows for full context (return codes, TXT
|
|
records, threat types, sample phishing URLs).
|
|
|
|
## Adding a new source
|
|
|
|
Every reputation backend implements the `Source` interface in its own
|
|
file and registers itself from `init()`. Skeleton:
|
|
|
|
```go
|
|
package checker
|
|
|
|
import (
|
|
"context"
|
|
sdk "git.happydns.org/checker-sdk-go/checker"
|
|
)
|
|
|
|
func init() { Register(&mySource{}) }
|
|
|
|
type mySource struct{}
|
|
|
|
func (*mySource) ID() string { return "mybl" }
|
|
func (*mySource) Name() string { return "My Blocklist" }
|
|
|
|
func (*mySource) Options() SourceOptions {
|
|
return SourceOptions{
|
|
Admin: []sdk.CheckerOptionField{ /* … */ },
|
|
}
|
|
}
|
|
|
|
func (*mySource) Query(ctx context.Context, domain, registered string, opts sdk.CheckerOptions) []SourceResult {
|
|
res := SourceResult{SourceID: "mybl", SourceName: "My Blocklist", Enabled: true}
|
|
// …populate Listed / Severity / Reasons / Evidence / Reference / Error
|
|
return []SourceResult{res}
|
|
}
|
|
|
|
func (*mySource) Diagnose(res SourceResult) Diagnosis {
|
|
return Diagnosis{Severity: SeverityCrit, Title: "Listed", Detail: "…"}
|
|
}
|
|
```
|
|
|
|
That's it: rules, the report, metrics, the standalone `/check` form
|
|
and the definition pick the new source up automatically. Sources that
|
|
need richer rendering (a per-vendor table, etc.) additionally
|
|
implement `RenderDetail(SourceResult) (template.HTML, error)`.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
make # standalone binary (HTTP server + /check form)
|
|
make plugin # checker-blacklist.so for happyDomain dynamic load
|
|
make docker # container image
|
|
```
|
|
|
|
## Running standalone
|
|
|
|
```bash
|
|
./checker-blacklist -listen :8080
|
|
# then GET /check, /definition, /health, …
|
|
```
|
|
|
|
The standalone binary embeds an `interactive` form on `GET /check` so a
|
|
human can paste a domain and run the full pipeline without happyDomain.
|
|
|
|
## License
|
|
|
|
MIT (checker code); Apache 2.0 SDK dependency.
|