# checker-blacklist

happyDomain checker that flags whether a domain is currently listed on
widely-used reputation systems.

## Sources

| Source                | Type            | API key needed | Configurable |
|-----------------------|-----------------|----------------|--------------|
| Spamhaus DBL          | DNS-based DBL   | no             | admin (default on) |
| SURBL multi           | DNS-based DBL   | no             | admin (default on) |
| URIBL multi           | DNS-based DBL   | no             | admin (default on) |
| NordSpam DBL          | DNS-based DBL   | no             | admin (default on) |
| SpamEatingMonkey Fresh| DNS-based DBL   | no             | admin (default on) |
| Tiopan DBL            | DNS-based DBL   | no             | admin (default on) |
| SORBS RHSBL           | DNS-based DBL   | no             | admin (default on) |
| Extra DNSBL zones     | DNS-based DBL   | no             | admin |
| Google Safe Browsing  | HTTPS lookup    | yes (admin)    | admin |
| OpenPhish public feed | downloaded list | no             | user (default on) |
| PhishTank             | downloaded list | no             | user (default on) |
| abuse.ch URLhaus      | HTTPS lookup    | free Auth-Key (admin) | user (default on) |
| abuse.ch ThreatFox    | HTTPS lookup    | free Auth-Key (admin) | user (default on) |
| abuse.ch MalwareBazaar| HTTPS lookup    | free Auth-Key (admin) | user (default on) |
| Botvrij.eu            | downloaded list | no             | user (default on) |
| Disconnect.me         | downloaded list | no             | user (default on) |
| OISD                  | downloaded list | no             | user (default on) |
| VirusTotal v3         | HTTPS lookup    | yes (admin)    | admin |
| AlienVault OTX        | HTTPS lookup    | free (admin)   | admin |
| Pulsedive             | HTTPS lookup    | free (admin)   | admin |
| Criminal IP           | HTTPS lookup    | yes (admin)    | admin |

### Obtaining API keys

**Google Safe Browsing** (option: `google_safe_browsing_api_key`)
1. Go to the [Google Cloud Console](https://console.cloud.google.com/) and create or select a project.
2. Enable the *Safe Browsing API* under *APIs & Services → Library*.
3. Create an API key under *APIs & Services → Credentials*.
4. The free tier allows up to 10 000 queries/day with no billing required.

**abuse.ch** (option: `urlhaus_auth_key` / `threatfox_auth_key` / `malwarebazaar_auth_key`)
1. Register a free account at [abuse.ch](https://abuse.ch/).
2. After login, retrieve your Auth-Key from your account profile page.
3. The same account and key works for URLhaus, ThreatFox, and MalwareBazaar — set it in each source option independently.
4. Free, no rate-limit tiers documented; the APIs are community-funded.

**VirusTotal** (option: `virustotal_api_key`)
1. Create a free account at [virustotal.com](https://www.virustotal.com/).
2. Go to your profile and copy the API key.
3. Free tier: 4 requests/minute, 500 requests/day. No billing required.
4. The public API key is sufficient; premium keys unlock higher quotas.

**AlienVault OTX** (option: `otx_api_key`)
1. Register a free account at [otx.alienvault.com](https://otx.alienvault.com/).
2. Go to *Settings → API Integration* to find your personal OTX key.
3. Free, no documented rate limits for the indicator lookup API.

**Pulsedive** (option: `pulsedive_api_key`)
1. Register a free account at [pulsedive.com](https://pulsedive.com/).
2. Go to your profile and copy the API key shown under *API*.
3. Free tier available; higher quotas with a paid plan.

**Criminal IP** (option: `criminal_ip_api_key`)
1. Register a free account at [criminalip.io](https://www.criminalip.io/).
2. Go to *My Information → API Key* to find your key.
3. Free tier: 100 requests/day. Paid plans unlock higher quotas.

DNS-based blocklists are queried in parallel. The OpenPhish feed is
downloaded once per hour by the provider and cached in memory.

## Common failure scenarios surfaced in the HTML report

The report opens with a diagnosis-first "Action required" section that
lists the most common, high-impact problems with a one-shot remediation:

1. **Listed on Spamhaus DBL / SURBL / URIBL / NordSpam / SpamEatingMonkey / Tiopan / SORBS**: direct lookup link and
   removal procedure URL per operator.
2. **Flagged by Google Safe Browsing**: link to Google Search Console's
   security-issues review request.
3. **Listed in the OpenPhish feed**: instructions to treat the host as
   compromised (audit recently-added files, rotate credentials), plus a
   link to OpenPhish feedback.
4. **Listed in URLhaus (active malware distribution)**: direct link to
   the abuse.ch reference page and per-URL takedown notification flow.
5. **VirusTotal multi-vendor flag**: Critical when at least one vendor
   reports `malicious`, Warning when only `suspicious`. Lists the
   flagging engines and links to the VT GUI page for re-scan / vendor
   contact.
6. **DNSBL query refused / API quota exhausted**: most public resolvers
   are blocked by DBL/URIBL operators; surfaced as a warning so it does
   not pollute the OK status.

A per-source detail table follows for full context (return codes, TXT
records, threat types, sample phishing URLs).

## Adding a new source

Every reputation backend implements the `Source` interface in its own
file and registers itself from `init()`. Skeleton:

```go
package checker

import (
    "context"
    sdk "git.happydns.org/checker-sdk-go/checker"
)

func init() { Register(&mySource{}) }

type mySource struct{}

func (*mySource) ID() string   { return "mybl" }
func (*mySource) Name() string { return "My Blocklist" }

func (*mySource) Options() SourceOptions {
    return SourceOptions{
        Admin: []sdk.CheckerOptionField{ /* … */ },
    }
}

func (*mySource) Query(ctx context.Context, domain, registered string, opts sdk.CheckerOptions) []SourceResult {
    res := SourceResult{SourceID: "mybl", SourceName: "My Blocklist", Enabled: true}
    // …populate Listed / Severity / Reasons / Evidence / Reference / Error
    return []SourceResult{res}
}

func (*mySource) Diagnose(res SourceResult) Diagnosis {
    return Diagnosis{Severity: SeverityCrit, Title: "Listed", Detail: "…"}
}
```

That's it: rules, the report, metrics, the standalone `/check` form
and the definition pick the new source up automatically. Sources that
need richer rendering (a per-vendor table, etc.) additionally
implement `RenderDetail(SourceResult) (template.HTML, error)`.

## Build

```bash
make                # standalone binary (HTTP server + /check form)
make plugin         # checker-blacklist.so for happyDomain dynamic load
make docker         # container image
```

## Running standalone

```bash
./checker-blacklist -listen :8080
# then GET /check, /definition, /health, …
```

The standalone binary embeds an `interactive` form on `GET /check` so a
human can paste a domain and run the full pipeline without happyDomain.

## License

MIT (checker code); Apache 2.0 SDK dependency.