Move the Svelte dashboard out of internal/web into its own top-level
frontend/ so it can be built and hosted independently, and make the
subidx binary API-only: drop the embedded dist and the catch-all /
handler, so unknown paths return a plain 404.
The frontend already knew how to live apart (VITE_API_BASE, -cors-origins,
vercel.json), so only the embed glue is gone. Also wire in Vercel Web
Analytics via @vercel/analytics.
Cross-origin dashboards could not read x-ratelimit-*, x-total-count,
x-max-seq or x-truncated, which silently disabled the budget meter and
live-feed cursor backfill on split deployments.
The lint job runs with checks:all, which enables ST1000 (each
package needs a doc comment). Give all six packages concise
package comments; validated locally with -checks all.
Drop unused test helper asn1MarshalOctetString (U1000) and the
vacuous uint64 >= 0 comparison in the readiness check (SA4003) —
readiness is simply that Total() answers without error.
Every fetched batch must now prove itself: before ingesting, the
tailer picks one cryptographically random entry, fetches its
get-proof-by-hash audit path, and verifies locally that the entry
hashes up to the currently verified STH root. Entries served outside
the signed tree — by a tampered connection or a misbehaving log —
fail the check, the batch is refused, the watermark does not advance,
and the cycle retries. Before this, entries were trusted purely on
TLS to the log endpoint; only the STH was verified.
The fake log in tests now serves a real RFC 6962 Merkle root and
audit paths, so the existing round-trip test exercises the gate, and
TestTamperedEntriesRejected proves that entries outside the signed
tree are never stored and never advance the watermark.
README updated: the provenance claim now matches what the code does.
Add LeafHash (SHA-256 of 0x00 || leaf per RFC 6962 §2), interior node
hashing, VerifyInclusion implementing the §2.1.2 audit-path algorithm,
and a ProofByHash client method for get-proof-by-hash (query properly
URL-escaped for base64 '+' and '/' bytes).
This is the machinery needed to bind fetched entries to a verified
signed tree head; the tailer wires it up in the next change.
Tests build a reference tree recursively (independent of the iterative
verifier) and cross-check every index for sizes 1..33, plus rejection
of wrong root, tampered/truncated paths, wrong index, foreign leaf.
statePriority omitted "frozen", so CurrentState() returned "" for
frozen logs and the tailer skipped them entirely — even though frozen
logs are readable, hold full history, and expose a final_tree_head,
which is exactly what the drain feature targets. Rank frozen between
pending and readonly, and give it the same drain-to-final-tree-head
treatment as readonly. Tests cover state recognition and priority.
The x509 path lowercased names, trimmed trailing dots and wildcard
prefixes, dropped control bytes, and deduplicated; the precert
(TBS) path returned raw SAN strings with none of that, leaving the
ingest filter single-layered for precerts and inconsistent casing in
stored names. Route both paths through the same normalization; test
covers mixed case, trailing dot, wildcard, and duplicate collapse.
VerifySTH accepted any root-hash length and ignored the DigitallySigned
hash/signature algorithm bytes, so a signed-but-degenerate STH (empty
root) or an algorithm-confused signature verified cleanly. Require a
32-byte root hash, a non-negative tree size, sha256 as the hash
algorithm, and a signature algorithm matching the pinned key type
(ecdsa=3, rsa=1).
Tests cover the valid case plus forged signature, short root, sha1
label, rsa label on an ecdsa key, and negative tree size — the
signature-verification failure path had no coverage before.
evictShard deleted arbitrary live buckets once a shard hit its key
cap. An attacker minting fresh keys (spoofed XFF, IPv6 /64 rotation)
could churn a shard until their own full bucket — or a victim's — was
evicted, resetting the counter and defeating the per-IP limit
entirely. Unknown keys are now denied while the shard is full;
expired buckets are still reclaimed first. Regression test pins the
fail-closed behavior and bucket survival for existing keys.
Endpoints come from externally fetched log lists and were used as-is:
any scheme was accepted and the default HTTP client followed up to 10
redirects, so a compromised list source could aim the tailer at
internal hosts (e.g. cloud metadata) even though STH signatures would
fail. Now NewClient requires https except for loopback (local test
servers), both clients refuse redirects outright, and a redirect
response surfaces as a normal HTTP error. Tests cover scheme
rejection and redirect refusal.
Log list entries joined as BaseURL+path break when the URL carries a
trailing slash: https://log.example.com/ + /ct/v1/get-sth becomes
...//ct/v1/get-sth and every request fails. Normalize once in
NewClient. Test covers the trailing-slash case end to end.
FetchAll issued three sequential 30s-timeout HTTP fetches with no
context, so shutdown could stall up to ~90s waiting for the hourly
sync loop. runTailers already owns a cancellation context; pass it
through so cancelation aborts in-flight list fetches immediately.
First tests for the package: context cancellation and error status.
A failed Watermark read (IO error, corruption, closed store) defaulted
the watermark to 0, silently re-draining the entire log from index 0.
Skip the log instead and log the failure.
Also make getRaw hold the store mutex and return errStoreClosed after
Close: pebble panics on Get-after-Close, so a tailer (or any reader)
racing shutdown crashed the process instead of receiving an error.
Test: TestWatermarkErrorSkipsLog proves the tailer never contacts the
log when the watermark read fails.
Top ran a quadratic selection sort over every apex, so 'subidx stats'
would effectively hang once the apex count reached the millions.
sort.Slice plus truncate is O(n log n); add a regression test for
ordering by count.
The API accepted any Host header, so a page running a DNS rebinding
attack could point attacker.com at 127.0.0.1 and read the collected
index from the victim's browser as same-origin JavaScript. Requests
whose Host is not localhost/127.0.0.1/::1 now get 421; -allowed-hosts
extends the list for exposed deployments. Rate limiting keyed on the
victim's own IP provided no protection here.
apply() sliced record values at fixed offsets and getMeta, Count and
Top called Uint64 without checking length, so a truncated or
hand-corrupted value panicked the ingest loop or the stats command.
All decoders now validate lengths; corrupt records are overwritten,
corrupt meta/count values return errors.
FetchAll swallowed per-source errors and always returned nil, so all
three upstreams failing was indistinguishable from an empty log list
and the hourly sync silently tailed nothing. Total failure now
returns an error; partial failure logs a warning.
Certificate SANs went into the store after only lowercasing, so a
unicode name like 'buecher.example.com' was stored as raw UTF-8 and
could never be found by search, which runs IDNA. Ingest now maps
names through a permissive IDNA profile (keeping DKIM-style
underscores that the strict lookup profile rejects), enforces RFC
1035 label lengths, and stores the canonical form. Normalize also no
longer lowercases before IDNA mapping, which produced wrong punycode
for decomposed unicode input.
fetchRange counted an entry as consumed before attempting to decode
it, so undecodable entries were skipped permanently with no record,
and the position could overshoot the requested range if a log served
more entries than asked. The position now advances by exactly the
number of received entries (clamped to the range) and skips are
logged with their index window.
Recount mutated totalBase/pendingTotal under the store mutex while the
ingest loop read and wrote them unlocked (a data race), and left stale
per-apex count caches that corrupted subsequent increments. Recount is
now an ingest-loop operation: it flushes pending records, rescans, and
resets loop-owned state inline, so counters have a single owner.
Every distinct client key allocated a bucket retained for up to 24
hours, and keys are cheap to mint at line rate (spoofed XFF values,
IPv6 /64s), so a remote attacker could grow the map until the process
ran out of memory. Shards now evict expired buckets and then arbitrary
victims once a shard exceeds 4096 buckets.
Scan materialized every record for an apex in memory, sorted it, and
the server rebuilt the full body twice more. One query for a large
apex could buffer hundreds of megabytes three times over; a few
concurrent requests exhausted RAM. Scan now keeps only the top N
results by insertion sequence via a bounded heap (default 100000,
configurable with -max-results), preserving collection order.
The limiter indexed the XFF chain at len(trustedHops) without checking
that the trusted suffix contained valid IPs. With an over-counted
trusted-proxy-hops setting, a single garbage header entry became the
rate limit bucket and attackers could rotate it to bypass the limit.
Bogus headers now fall back to the connection address.
VerifySTH silently passed when no log key was pinned or when an STH
arrived without a signature, and a failed verification only produced a
log line while ingestion continued from the unauthenticated source.
NewClient now refuses logs without keys, the tailer skips the fetch
cycle on verification failure, and it rejects list entries whose
log_id does not match SHA-256 of the key.
NewClient returns (nil, err) when a log list entry has an undecodable
key, but the error path called client.ShortID() on the nil client,
panicking the tailer goroutine and crashing the process.
Both call sites pass the extnValue OCTET STRING contents, which per
RFC 6962 section 3.3 are already the TLS-encoded SCT list. The extra
asn1.Unmarshal failed on virtually every real certificate (the first
byte is the high byte of the uint16 length), so embedded SCT
timestamps were silently dropped and first_seen fell back to the leaf
timestamp. The test masked this by double-wrapping the value.
Certificate SANs are attacker-controlled and were copied into store
keys unfiltered. Since 0x00 is the apex/sub key separator, a SAN like
'ab\0cd.qqq' corrupted per-apex counts during recount and made scan
bounds ambiguous. Names now must be printable ASCII at ingest, and
Normalize rejects them as defense in depth.
The tailer fsynced the watermark immediately after pushing records into
the async ingest batch, so a crash could persist the marker without the
data it covers, losing those entries permanently. Watermark updates now
travel through the same channel as records and are written inside the
same Pebble batch, and never move backwards.
