Security: Residual SSRF on InferenceUrl — DNS/rebind + validator redirect (incomplete fix after #505/#534) #1470
Summary
Residual SSRF after prior paid mitigations (v0.2.7 / v0.2.8):
| Prior fix | What it covered | What remains broken |
|---|---|---|
| PR #534 | TA client no redirects (NewNoRedirectClient) |
Direct dial to DNS→private IP still allowed |
| PR #505 | ValidateURLWithSSRFProtection rejects literal private IPs |
No DNS resolve; hostnames always pass |
Any participant-controlled InferenceUrl hostname (or public URL that redirects) can force validator DAPIs (and other sinks) to connect to loopback, link-local metadata (169.254.169.254), or RFC1918 during mandatory payload retrieval / PoC validation / TA executor forwarding.
This is incomplete remediation, not a greenfield finding.
Severity (self-assessment)
High — protocol-mandated HTTP from validator hosts to attacker-controlled URL; host infrastructure compromise (IMDS, localhost admin, internal network). Not direct on-chain fund drain.
Architecture / threat model
Attacker sets InferenceUrl (hostname / rebind / open redirect)
│
▼
Chain: Participant.InferenceUrl
│
┌────┴─────────────────┬──────────────────┐
▼ ▼ ▼
Validator payload TA → executor PoC off-chain
retrieval (primary) POST proof HTTP
DEFAULT http.Client NoRedirectClient multi-worker
FOLLOWS REDIRECTS still dials DNS IP
Attacker preconditions
- Can set/update
InferenceUrlviaMsgSubmitNewParticipant/ unfunded variant whileOpenRegistrationPermissionallows, or - Already has a hostname URL on-chain → DNS rebinding needs no new tx
Victim: any honest host running DAPI that validates / TAs / PoC-validates against that participant.
Root cause (registration gate)
inference-chain/x/inference/utils/signature_and_url_validation.go:
func ValidateURLWithSSRFProtection(fieldName, raw string) error {
// scheme http/https + host non-empty
if isLocalhost(host) { return err }
ip := net.ParseIP(host)
if ip != nil {
if isPrivateIP(ip) { return err }
}
return nil // hostname path: NEVER resolves DNS
}
Called from:
types/message_submit_new_participant.goValidateBasictypes/message_submit_new_unfunded_participant.goValidateBasic
Unit tests only cover: public URL, localhost, 192.168.0.1 — no hostname/DNS cases
(signature_and_url_validation_test.go).
Matrix (registration)
| URL | Gate result | Notes |
|---|---|---|
http://127.0.0.1/ |
REJECT | literal |
http://169.254.169.254/ |
REJECT | literal |
http://localtest.me/ |
ACCEPT | DNS → 127.0.0.1 |
http://ssrf.attacker.tld/ |
ACCEPT | attacker DNS |
http://0x7f000001/ / decimal IP strings |
ACCEPT* | ParseIP fails → hostname |
http://[::ffff:127.0.0.1]/ |
REJECT | mapped IPv6 parsed |
*Depends on OS resolver if those “hostnames” resolve.
URL write path
keeper/msg_server_submit_new_participant.go:
- Create participant with
Url - Update existing:
existing.InferenceUrl = msg.Url(same permission gate)
DAPI registers PublicUrl via participant_registration.go.
If registration later closes, create/update txs fail — rebinding still works for hostnames already stored.
HTTP sinks (all participant InferenceUrl)
1) Validator payload retrieval — primary
validateInferenceAndSendValMessage
→ retrievePayloadsWithRetry // up to 10 attempts, long backoff
→ RetrievePayloadsFromExecutor
→ Participant.InferenceUrl from chain
→ BuildPayloadRequestURL(..., "v1/inference/payloads", id)
→ FetchPayloadsHTTP(payloadRetrievalClient, ...)
// decentralized-api/internal/validation/payload_retrieval.go
var payloadRetrievalClient = &http.Client{
Timeout: 30 * time.Second,
// no CheckRedirect → default Go client follows up to 10 redirects
// no DialContext IP ACL
}
Also sends validator identity headers (X-Validator-Address, signature, epoch) to whatever is dialed.
2) Transfer Agent → executor
getExecutorForRequest → GetRandomExecutor → executor.InferenceUrl
POST via NewNoRedirectClient // redirects blocked (#534)
Direct DNS → private IP still dials.
3) PoC off-chain validator
decentralized-api/poc/validator.go — loads InferenceUrl, parallel proof HTTP workers.
4) Devshard
Reuses FetchPayloadsHTTP / BuildPayloadRequestURL.
Not a fix: proxy sidecar
proxy/sidecar resolves DNS and skips private IPs only when building nginx whitelist — does not stop DAPI egress SSRF.
Attack chains
A — DNS hostname (register/update)
- Point
ssrf.attacker.tld→169.254.169.254/ internal IP / loopback MsgSubmitNewParticipant{ Url: "http://ssrf.attacker.tld" }- Passes
ValidateBasic - When peers validate / TA / PoC-hit this host → SSRF
B — DNS rebinding (no chain update)
- Register hostname with public A record
- During validation window, flip A to private (low TTL)
- No on-chain re-validation at fetch time
C — Redirect residual (validator payload client only)
InferenceUrl = http://attacker.example/open(public, accepts registration)- Attacker responds
302 Location: http://127.0.0.1:9200/admin/v1/config(or IMDS) - TA client will not follow;
payloadRetrievalClientwill - Can turn this into localhost admin/config impact on validators without publishing admin ports
Lab proof (safe, local)
Using public DNS name localtest.me → 127.0.0.1:
http://127.0.0.1:PORT/...→ REJECT (literal)http://localtest.me:PORT/...→ ACCEPT (hostname)- Validator-style HTTP GET retrieves loopback service body (exfiltrates lab secret)
Runnable replica (research package): registration gate replica + e2e fetch against local victim service.
Impact
| Impact | Detail |
|---|---|
| Confidentiality | Cloud IMDS, internal HTTP, localhost admin/config on validators |
| Integrity | Pivot into host networks; secondary compromise of DAPI/node |
| Availability | Hang/slow validators on attacker-controlled responses; retry amplification (×10) |
| Scale | Every honest validator that retrieves payloads becomes an SSRF client |
Required remediation
- Dial-time deny-list (mandatory): shared safe
http.Clientthat resolves, rejects loopback/link-local/RFC1918/ULA, dials only public IPs - Apply to all sinks:
payloadRetrievalClient, PoC proof client, TA client (defense-in-depth) - Disable redirects on payload client or re-check
Locationhost/IPs with the same policy - Registration-time resolve is optional extra; not sufficient alone (rebinding)
- Regression tests: hostname→127.0.0.1, hostname→169.254.169.254, public→302 private, IPv4-mapped IPv6, decimal/hex host strings
Sketch
// DialContext: LookupIP → reject isPrivateIP → dial public only
// CheckRedirect: ErrUseLastResponse OR validate Location with same policy
Evidence index
| Path | Role |
|---|---|
inference-chain/x/inference/utils/signature_and_url_validation.go |
Gate gap |
inference-chain/x/inference/utils/signature_and_url_validation_test.go |
Thin tests |
inference-chain/x/inference/types/message_submit_new_participant.go |
ValidateBasic |
inference-chain/x/inference/keeper/msg_server_submit_new_participant.go |
Create/update URL |
decentralized-api/internal/validation/payload_retrieval.go |
Client + sink |
decentralized-api/internal/validation/inference_validation.go |
Retry loop |
decentralized-api/internal/server/public/post_chat_handler.go |
TA NoRedirect + executor URL |
decentralized-api/poc/validator.go |
PoC sink |
inference-chain/app/upgrades/v0_2_8/upgrades.go |
Prior bounty / PR #505 #534 context |
Disclosure
- Static analysis + local lab only; no production hosts targeted.
- If public issues are not preferred, please close and accept via HackerOne (gonka.ai report vulnerability); we can re-file privately with PoC artifacts.
💬 Comments (2)
I'd like to pick this up. The residual-SSRF analysis matches what I see in the code — the real fix has to be at dial-time (registration-time DNS resolution alone can't stop rebinding), via a shared SSRF-safe HTTP client applied to all participant-InferenceUrl sinks, plus disabling redirects on the payload retrieval client.
@tcharchian could you assign this to me?
🔄 Auto-synced from Issue #1470 every hour.
Deep dive (follow-up analysis)
Historical context (residual after prior SSRF fixes)
From
inference-chain/app/upgrades/v0_2_8/upgrades.gobounty notes:NewNoRedirectClient).ValidateURLWithSSRFProtectionforInferenceUrl(literal private IPs + timeouts).This report is a residual / incomplete-fix issue, not a greenfield class:
CheckRedirect→ErrUseLastResponse)payloadRetrievalClienthttp.Client→ follows redirectsFull sink map (participant
InferenceUrl→ HTTP)Validator payload retrieval (primary)
validateInferenceAndSendValMessage→retrievePayloadsWithRetry(up to 10 attempts) →RetrievePayloadsFromExecutor→BuildPayloadRequestURL(InferenceUrl, "v1/inference/payloads", id)→FetchPayloadsHTTPClient:
payloadRetrievalClient = &http.Client{Timeout: 30s}— noCheckRedirect, no dial ACL.Transfer Agent → executor
getExecutorForRequest→executor.InferenceUrl→POSTviaNewNoRedirectClientRedirects blocked; direct DNS→private IP still dials.
PoC off-chain validator
poc/validator.goloadsParticipant.InferenceUrland hits it via proof HTTP client (parallel workers).Devshard paths reuse
FetchPayloadsHTTP/BuildPayloadRequestURL.Proxy sidecar does resolve DNS and skips private IPs — but only for nginx whitelist generation, not for DAPI egress.
Attack chains
A. DNS hostname (registration-time)
MsgSubmitNewParticipant{Url: "http://ssrf.attacker.tld"}with A/AAAA→169.254.169.254/ RFC1918 / loopback.ValidateBasicaccepts becausenet.ParseIP(hostname) == nil.B. DNS rebinding (no chain update)
Register hostname with public A record; later flip A to private during validation windows. No re-check at fetch.
C. Redirect residual (validator only)
Public
InferenceUrlreturns302 Location: http://127.0.0.1:9200/admin/v1/config(or IMDS).TA client will not follow;
payloadRetrievalClientwill follow → can turn #1470 into localhost admin/config impact on validators.Control-flow gap (source)
Unit tests only cover public URL, localhost, and
192.168.0.1— no hostname/DNS cases.URL write path
Existing participants can update
InferenceUrlvia the sameMsgSubmitNewParticipant(whenOpenRegistrationPermissionallows).Even if registration is later closed, rebinding still works for already-stored hostnames.
Lab proof (safe)
localtest.me→127.0.0.1:http://127.0.0.1:...→ REJECThttp://localtest.me:...→ ACCEPTRequired fix (defense-in-depth)
http.Client.Locationhost/IPs.Happy to provide a draft patch or HackerOne-formatted write-up if public issues are not the preferred channel.