For reading and verification only. You may not copy, use, modify or redistribute this code. Full notice on the main page.
tunnel/doc.go
| 1 | // Command all-tunnel is the SNI-routing tunnel node of ALL 2.0 end-to-end encryption (the design plan, sections 2, 3 and 5, and |
| 2 | // the contract document, which this program implements). |
| 3 | // |
| 4 | // What it does: ChatGPT (any HTTPS client) connects to :443 and speaks normal TLS. The node reads only the cleartext ClientHello to learn the SNI name, |
| 5 | // finds the device that owns that name, and copies raw bytes between the client and the device. The TLS server is the DEVICE (it holds the private |
| 6 | // key and the publicly trusted certificate for its own name); this program never decrypts, never terminates and never rewrites a client's TLS, and it |
| 7 | // never logs payload bytes, addresses or names (only times, sizes, reasons and stream ids). |
| 8 | // |
| 9 | // # Ports |
| 10 | // |
| 11 | // There are exactly two, and no other: |
| 12 | // |
| 13 | // :443 the one public listener. The first thing read is the ClientHello's SNI: |
| 14 | // - the node's OWN name (-node-host): the control handler. The device speaks TLS 1.3 to it, with the node's public-CA certificate |
| 15 | // and ALPN all-tunnel/2, and protocol v2 inside (control and data connections, below); |
| 16 | // - any other name: a client of a device. If a device is online for that name and may serve, the node sends it OPEN and splices the |
| 17 | // client's bytes to the data connection that comes back; otherwise the client gets a TLS alert and nothing else happens. |
| 18 | // :80 the node's own ACME webroot and nothing else: GET/HEAD /.well-known/acme-challenge/<token> for the node's own name, from |
| 19 | // -acme-webroot. It never reaches a device (a certificate for a device is issued over DNS-01, and CAA allows nothing else). |
| 20 | // |
| 21 | // The 1.1 experiment's separate device port and the self-signed certificate that devices pinned are gone. |
| 22 | // |
| 23 | // # Device protocol (version 2) |
| 24 | // |
| 25 | // Every connection to the node's own name first completes TLS, then the device sends ONE text line (ASCII 0x20-0x7e, fields separated by one space, |
| 26 | // ended by LF, at most 8192 bytes, within first_line_timeout_s). The grammar is contracts/schemas/wire-v2.grammar.json; proto.go implements it and |
| 27 | // protocol_test.go runs every golden vector through it. The first line tells the two kinds of connection apart. |
| 28 | // |
| 29 | // Control connection (one per device, kept open): |
| 30 | // |
| 31 | // device -> node HELLO 2 <session> the device's session JWT (aud all-tunnel, 300 s); no host name |
| 32 | // node -> device CHALLENGE <nonce> once the session verified OFFLINE against the control plane's keys |
| 33 | // device -> node PROOF <dpop> a DPoP proof over the NODE's URL (htu), the nonce and the session, signed by the session's bound key |
| 34 | // node -> device OK <json> after the node asked the control plane (authorize): host, status, valid_until, windows, relayed as given |
| 35 | // or ERR <code> a refused HELLO closes the connection |
| 36 | // |
| 37 | // The host a device is served under comes ONLY from the control plane's answer; nothing the device says can name one. The connection lives until the |
| 38 | // answer's valid_until: before that moment minus renew_before_valid_until_s the device sends |
| 39 | // |
| 40 | // device -> node RENEW <new session> the same subject; CHALLENGE and PROOF follow, then the node calls authorize AGAIN and answers |
| 41 | // node -> device OK <json> with the NEW valid_until and status (a trial that ended is gated from here, with no event needed) |
| 42 | // |
| 43 | // and a RENEW that is refused keeps the connection with its old valid_until. Without a RENEW the node sends CLOSE expired at valid_until and closes. A |
| 44 | // second authenticated HELLO of the same subject replaces the first (CLOSE replaced; its open streams keep running); a HELLO whose answer names a host |
| 45 | // that another subject holds is refused, and a RENEW whose answer names another host closes the connection (CLOSE revoked): the old host stops being |
| 46 | // served at once. Status paused keeps the connection but opens no stream; gated opens them (held to the day's byte and stream budget, see |
| 47 | // Hardening). PING is answered with PONG; a connection silent for dead_peer_s is dropped (CLOSE idle). HINT carries source addresses the device |
| 48 | // marked good (the node's hint table, and Config.OnHint). |
| 49 | // |
| 50 | // OPEN: |
| 51 | // |
| 52 | // node -> device OPEN <stream-id> <attach-secret> <peer-ip> <class> |
| 53 | // |
| 54 | // is sent ONLY after the client's ClientHello is complete and well formed (a truncated, stalled, oversized or malformed one, or one without a usable |
| 55 | // SNI, never produces an OPEN), never for a paused device, never at or after valid_until. stream-id is 128 random bits, attach-secret is 256 random |
| 56 | // bits and valid for ONE ATTACH, peer-ip is the client's address (no zone) and class is openai (the address is in OpenAI's published list), hint (the device marked it with |
| 57 | // HINT) or anon (Config.Classify overrides this). |
| 58 | // |
| 59 | // Data connection (one per OPEN, dialed by the device to the same name and port): |
| 60 | // |
| 61 | // device -> node ATTACH <stream-id> <attach-secret> |
| 62 | // node -> device ERR auth | ERR unknown_stream | ERR bad_attach (then close), or NOTHING: |
| 63 | // |
| 64 | // on success there is NO reply line. The very next byte the node sends is the first byte of the raw stream of ONE client connection (the ClientHello, |
| 65 | // replayed exactly as received), and every byte the device sends from then on goes to that client. A wrong secret is refused and does not consume the |
| 66 | // stream; a used or unknown id is unknown_stream. EOF in one direction is propagated as a half-close and the other direction then has half_close_s to |
| 67 | // finish; each stream is bounded by the idle timeout, the maximum age and a byte cap, and streams per host and in total are capped. |
| 68 | // |
| 69 | // # Trust and what the node knows |
| 70 | // |
| 71 | // The node trusts the control plane's keys (a KeySource) to verify sessions and the control plane's answer (a ControlPlane) to decide what a device |
| 72 | // may do. The production build wires the real ones (backend_prod.go); the test-node build wires fakes. Every number, window and pattern is read from |
| 73 | // policy.json (a byte-identical copy of contracts/policy.json that the contract generator's sync writes). |
| 74 | // |
| 75 | // # The control plane (controlplane.go, jwks.go, feed.go) |
| 76 | // |
| 77 | // The node holds ONE secret: its own Ed25519 signing key (-node-key-file, mode 0600; -node-key-init makes it and prints the TUNNEL_NODE_KEYS entry that |
| 78 | // registers it, removing the kid from that variable revokes the node). Every request to /v1/tunnel/* carries an assertion in x-all-node-assertion, a |
| 79 | // 60 s JWS bound to the method, the exact URL and the SHA-256 of the exact body, used once. Nothing else authenticates the node: no shared secret, no |
| 80 | // commerce key. The client reads only the configured control plane (never a redirect, never a URL an answer names), caps every answer, and turns |
| 81 | // whatever it cannot read into bad_response: no text from the network reaches a device or a log. |
| 82 | // |
| 83 | // authorize asked at every HELLO and at every RENEW (the end of a trial turns a connection into gated with no event); at most |
| 84 | // authorization.authorize_max_concurrent calls run at once and what waits is bounded, the rest is told control_unavailable |
| 85 | // JWKS the published keys: fetched at start and every cache_s, reloaded when a session names an unknown kid (one fetch per minimum gap, |
| 86 | // however many sessions ask), replaced whole on every fetch, at most max_keys; after stale_fail_closed_s without a refresh the node |
| 87 | // vouches for nothing: new HELLOs and RENEWs are answered control_unavailable (not auth), connections already up are untouched |
| 88 | // feed GET /v1/tunnel/feed with a held poll (wait), applied within timing.feed_apply_deadline_s: revoke, release and license_end close the |
| 89 | // connection (CLOSE revoked/released) and kill the streams of the host; pause kills the streams, stops new ones and forwards the |
| 90 | // device's signed statement; resume asks the control plane again (its answer decides); kill/kill_clear are the global switch; reset |
| 91 | // closes everything so that devices come back and are authorized again. Events are applied once, in epoch order; a duplicate, a |
| 92 | // replay or an event older than a connection's last answer changes nothing |
| 93 | // STATEMENT the access statement of an authorize answer, forwarded unchanged right after the OK (under the write lock: no OPEN comes before it) |
| 94 | // |
| 95 | // # When the control plane does not answer |
| 96 | // |
| 97 | // What is authenticated goes on until valid_until: nothing is asked of the control plane to keep a connection or to open a stream. A new HELLO needs |
| 98 | // a good authorize. A PAID connection whose valid_until arrives while the control plane is not answering (its own RENEW got a 5xx, a 429 or a |
| 99 | // timeout, or nothing has answered for two poll periods) is kept for authorization.grace_paid_s more, opens no new stream in that time, and is closed |
| 100 | // when it is over; a trial or a gated connection is closed at valid_until, and so is a paid one whose RENEW the control plane answered (it said no). |
| 101 | // A connection that ends because its grant ended (expired, revoked, released, killed) takes the streams of its host with it. |
| 102 | // |
| 103 | // # Draining (drain.go) |
| 104 | // |
| 105 | // SIGTERM stops new ClientHellos (a TLS alert, no OPEN) and new HELLOs (ERR control_unavailable), tells every device DRAIN <quiet_s> (prefer the other |
| 106 | // node), keeps serving the streams that are open, the ATTACH of streams that already have their OPEN and the RENEW of the devices that hold them, and |
| 107 | // exits when the streams are gone or timing.drain_quiet_s (20 minutes, -drain-quiet) is over, whichever comes first, with CLOSE drain to the devices. |
| 108 | // A second SIGTERM stops waiting; SIGINT stops at once; SIGHUP reads the node's certificate again (certreload.go also looks at the files on its own). |
| 109 | // |
| 110 | // # Hardening (limits.go, hints.go, gated.go, openai_prefixes.go, observer.go, metrics.go; deploy/) |
| 111 | // |
| 112 | // Every number is policy.node_limits or policy.gated; nothing in this program types one. A SOURCE is an IPv4 address, or the /64 of an IPv6 one. |
| 113 | // |
| 114 | // at accept a source that is banned, that opens connections faster than per_ip_new_per_s, or that holds per_ip_concurrent already (pre-hello, and |
| 115 | // the client streams it has open: a device's control connection stops counting when it authenticates, a data connection when it |
| 116 | // attaches) is dropped with no answer; an address of the OpenAI list has openai_per_ip_factor times the limits and is never banned; |
| 117 | // an address that holds an authenticated control connection has device_per_ip_factor times the limits (a device opens one data |
| 118 | // connection per stream it serves), and a connection that has authenticated or attached gives back the token it took from the rate |
| 119 | // bucket; :80 is under the same limits |
| 120 | // ClientHello 5 s (client_hello_deadline_s), no OPEN before it is complete and well formed |
| 121 | // unknown names a source that tries more than unknown_sni_per_ip DISTINCT names that are not a device online (or sends no TLS) within |
| 122 | // unknown_sni_window_s is banned for unknown_sni_ban_s, in memory; the same name again costs nothing (ChatGPT retrying an offline device |
| 123 | // is not a scan); an address a device marked with HINT is never struck, and the shared overflow entry (what sources get when the |
| 124 | // table is full) is never struck or banned, so a scanner cannot turn a full table into a ban of every new source |
| 125 | // per device label_streams_max streams: label_anon_slice for any source, and the rest only for sources the device marked with HINT (a marked source |
| 126 | // may also use free anonymous room); new streams from sources that are not marked are paced at label_new_per_s (burst label_new_burst) |
| 127 | // global global_streams_max, of which openai_reserved_pct percent only openai or hint sources may use (a quota, not an allowlist: the class |
| 128 | // comes from OpenAI's published list, read at start and every openai_prefix_refresh_s, kept in a file, and a list that is empty, corrupt, |
| 129 | // too big or has a prefix wider than the policy's minimum replaces nothing) |
| 130 | // HINT addresses a device says just completed authenticated calls: memory only, hint_ttl_s from the last HINT, at most |
| 131 | // hint_addresses_per_label per device, filed under the host the control plane gave THAT connection |
| 132 | // gated gated.bytes_per_day and gated.streams_per_day per device host and UTC day, in memory; counted on every status the control plane gives |
| 133 | // (authorize at HELLO and RENEW, and the refresh after a pause or resume), so a device that turns gated mid-connection is cut on its |
| 134 | // open streams when the bytes run out, and reconnecting does not buy a new budget. The budget is spent by every stream of the host, |
| 135 | // whoever opens it: a device's name is public (certificate transparency), so a third party can use up a gated device's day (200 |
| 136 | // streams or 10 MB) and keep its paywall card from reaching the user until midnight UTC. Accepted: a gated device is not a service |
| 137 | // anyone depends on, and the price of the alternative (telling clients apart) is a rule that fails open |
| 138 | // sockets TCP keepalive from tcp_keepalive_s on everything the node accepts; idle and maximum age of a stream are timing.stream_idle_s and |
| 139 | // timing.stream_max_age_s |
| 140 | // logs no address, no name, no label, no device id in a log line, ever: a line that must tell two sources apart carries Server.short, the |
| 141 | // per-boot salted short hash; lines a flood can repeat are written at most once a second per reason, with how many were left out |
| 142 | // observer the cleartext start of a handshake, counted by a closed list of words: the groups a ClientHello offers (X25519MLKEM768 included) and |
| 143 | // whether it carries an ECH extension, the version, cipher and group the device's ServerHello selects, HelloRetryRequests. Nothing |
| 144 | // else, nothing that names a client |
| 145 | // /metrics -metrics-addr (loopback only, 127.0.0.1:9101 in the production build) serves /metrics and /healthz: aggregate counters, no series |
| 146 | // per device, label, address or name, the node certificate's not_after and the age of the control plane's keys included; a metrics |
| 147 | // port that cannot be bound is a log line (metrics_unavailable), not a node that does not start |
| 148 | // kernel deploy/nftables.conf and deploy/sysctl.conf: per-source connection and rate limits above the node's own, SYN cookies. The kernel |
| 149 | // counts every connection a source HOLDS, a device's data connections (one per open stream) included, while the node frees its own |
| 150 | // slot at attach: the held limit is sized for that |
| 151 | // |
| 152 | // # Test node |
| 153 | // |
| 154 | // `go build -tags testnode` (build.sh test-node) builds the same node with file-driven fakes for the keys and the control plane. It is what |
| 155 | // ALL_TEST_NODE_BIN names for the device's tests; see testnode.go. It is never deployed. |
| 156 | // With -test-tap FILE it also records every byte it forwards and every control line for the blindness gate (tap_test_hook.go); the build with |
| 157 | // `-tags "testnode blindmutant"` (build.sh blind-mutant) is the node that gate must catch (blindmutant_test_hook.go). Neither is ever deployed. |
| 158 | // |
| 159 | // # Run |
| 160 | // |
| 161 | // all-tunnel -node-id t1 -node-kid node-t1-1 -node-host t1.nodes.example.org -node-key-file node.key \ |
| 162 | // -cert-file fullchain.pem -key-file privkey.pem -acme-webroot /var/lib/all-tunnel/acme |
| 163 | // |
| 164 | // See main.go and backend_prod.go for every flag; each also has a TUNNEL_* environment variable. |
| 165 | // |
| 166 | // # Operating it |
| 167 | // |
| 168 | // deploy/ holds what a node needs around the binary (the unit, the certbot units and hooks, the volatile journal, nftables, sysctl). the deploy tool |
| 169 | // installs it and runs the whole procedure (provision, bootstrap, canary-first deploy with its DNS, stream and health gates, rollback); the canary tool is |
| 170 | // the canary device the monitors probe through each node; the node runbook is the procedure, drain and lost-call caveat included. |
| 171 | package main |