mirror of
https://github.com/vee1e/subidx.git
synced 2026-09-01 09:50:18 +00:00
lakshit verma
41a2828ef0
Move the Svelte dashboard out of internal/web into its own top-level frontend/ so it can be built and hosted independently, and make the subidx binary API-only: drop the embedded dist and the catch-all / handler, so unknown paths return a plain 404. The frontend already knew how to live apart (VITE_API_BASE, -cors-origins, vercel.json), so only the embed glue is gone. Also wire in Vercel Web Analytics via @vercel/analytics.
101 lines
6 KiB
Markdown
101 lines
6 KiB
Markdown
# subidx
|
|
|
|
Self-hosted passive subdomain enumeration from Certificate Transparency. subidx tails public CT logs (the records of every HTTPS certificate issued), indexes every hostname it sees, and serves them through a search API you control. Think crt.name, but the data lives on your machine.
|
|
|
|
|
|
|
|
## Quick start
|
|
|
|
```
|
|
go build -o subidx .
|
|
./subidx tail -store ./data # collect names (runs forever, Ctrl-C to stop)
|
|
./subidx serve -store ./data -addr :8099 # search what you collected
|
|
```
|
|
|
|
```
|
|
curl "http://localhost:8099/v1/search?apex=letsencrypt.org"
|
|
```
|
|
|
|
The binary is API-only. The dashboard is a separate SPA in `frontend/` (Vite + Svelte): search any apex, filter and sort results instantly, toggle first-seen dates, export txt/csv, and watch a live feed of newly collected names. Deploy it to any static host (Vercel, Render, or your own), pointing it at the API with `VITE_API_BASE` and allowing its origin with `-cors-origins`. For local work, `make dev` runs it against a local `subidx serve`.
|
|
|
|
## What gets stored
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `apex` | registered domain, e.g. `example.com` |
|
|
| `sub` | full subdomain, kept whole (`ap.www.sandbox.namecheap.com` is one record under `namecheap.com`) |
|
|
| `first_seen` | earliest date it appeared in any watched log |
|
|
|
|
## Why not just use subfinder?
|
|
|
|
Tools like [subfinder](https://github.com/projectdiscovery/subfinder) query other people's services at run time and inherit their quotas, captchas, and blind spots, and every run starts from zero. subidx pulls from the source once and owns the index:
|
|
|
|
- **Continuous coverage.** The tailer runs 24/7 with crash-safe resume; a certificate issued thirty seconds ago is already searchable.
|
|
- **First-seen timelines.** Every name carries its earliest seen date (`&dates=1`). Diff over time to catch new subdomains between engagements.
|
|
- **No keys, no quotas.** Your only dependencies are the log lists themselves.
|
|
- **Verified provenance.** Log keys are pinned against the log lists, tree heads are cryptographically verified, and each fetched batch must pass an RFC 6962 inclusion-proof spot check before it is stored.
|
|
- **Historical drains.** Years of history from retired logs can be backfilled, if you have the terabytes.
|
|
- **An API and dashboard, not just a CLI.** Pipe results into your tooling, or browse them in the separately hosted UI.
|
|
|
|
Limitations: CT only sees names that were issued a certificate. Names that live only in DNS are invisible here, which is why subidx complements rather than replaces broader-source tools.
|
|
|
|
## How it works
|
|
|
|
1. **Discovery.** Every hour, subidx fetches Chrome's log lists (plus Apple's) and merges them. Live logs are tailed; old and rejected logs hold the drainable history (`-no-drain` to skip).
|
|
2. **Tailing.** Each log is polled every few seconds for new entries, up to 1000 per request, with a per-log watermark that never moves backwards across restarts.
|
|
3. **Parsing.** Entries are decoded just far enough to read the SAN list; everything else is discarded.
|
|
4. **Normalizing.** Names are lowercased, wildcards and trailing dots stripped, reserved names rejected, and each name assigned to its registered domain (apex) via the Public Suffix List.
|
|
5. **Storing.** A single writer inserts into an embedded Pebble store. Duplicates keep only the earliest date; nothing is deleted.
|
|
6. **Serving.** An HTTP API with rate limiting and health checks. The dashboard is a separate frontend built from `frontend/` and served by any static host.
|
|
|
|
## Commands
|
|
|
|
| Command | What it does |
|
|
|---|---|
|
|
| `tail` | Watch CT logs and store names. Runs until you stop it. |
|
|
| `serve` | Serve the read-only search API and health endpoints. Add `-no-tail` to serve without collecting. |
|
|
| `stats` | Print total records and the top 10 domains. `-recount` fixes drifted counters. |
|
|
| `version` | Print the version. |
|
|
|
|
Useful flags:
|
|
|
|
| Flag | Default | Meaning |
|
|
|---|---|---|
|
|
| `-store` | `./data` | Where the database lives |
|
|
| `-addr` | `127.0.0.1:8080` | Listen address (binds localhost by default; use `:8080` to expose) |
|
|
| `-poll-interval` | `3s` | How often each log is checked for new entries |
|
|
| `-window` | `512` | Entries fetched per request while catching up |
|
|
| `-no-drain` | off | Skip old and rejected logs (years of history, terabytes). Works on `tail` and `serve` |
|
|
| `-rate-limit` | `1000` | Search requests allowed per IP per rolling 24 hours |
|
|
| `-max-results` | `100000` | Max results buffered per search query |
|
|
| `-allowed-hosts` | loopback names | Host header values to accept (blocks DNS rebinding; add your hostname when exposing) |
|
|
| `-cors-origins` | empty | Browser origins allowed to call the API from a separately hosted frontend |
|
|
| `-trusted-proxy-hops` | `0` | Proxies in front of you; 0 means X-Forwarded-For is ignored |
|
|
|
|
Only one process can use a store directory at a time; the database takes an exclusive lock.
|
|
|
|
## The API
|
|
|
|
`GET /v1/search?apex=example.com` returns one name per line:
|
|
|
|
| Case | Response |
|
|
|---|---|
|
|
| Known domain | `200`, one name per line |
|
|
| Unknown domain | `200`, empty body (not 404) |
|
|
| Not a bare domain | `400`, plain text reason |
|
|
| Missing `apex` | `400`, `missing apex parameter` |
|
|
| `&dates=1` | Tab plus first-seen date per line |
|
|
| `&format=json` | JSON array of `{"sub":"..."}` objects |
|
|
| `&format=ndjson` | One JSON object per line, flushed for streaming clients |
|
|
| HEAD | `405` |
|
|
|
|
Search responses carry `x-total-count` (names in the index for the apex) and `x-max-seq` (a cursor for change tracking). Search and stats are gzipped on request.
|
|
|
|
Live dashboards get two more endpoints:
|
|
|
|
- `GET /v1/watch?apex=X&after=N`, NDJSON of names collected since sequence `N`, with `x-max-seq` and `x-truncated`. A cheap "what changed" poll.
|
|
- `GET /v1/feed?apex=X`, server-sent-event stream of new names for that apex, one rate-budget unit per session. Slow consumers get a `resync` event instead of silent gaps.
|
|
|
|
Also: `/v1/stats` (`{"total":N,"top":[...]}`, optional `&n=`, cached, top-k bounded), `/healthz`, `/readyz`. Health endpoints skip the rate limit; everything else is counted.
|
|
|
|
|