logs.gokuls.in

Problem

Only a message's first send is handed to mesh neighbours when no carrier reaches the recipient: handle_send_failure offers it when no carrier took the frame, and handle_send_success offers it when a carrier took it for a recipient it cannot reach. The resend paths asked neither question:

• the retry loop in process_retry_queue (protocol/mod.rs), both branches;
• try_flush_send_via, behind flush_outbox_all and flush_outbox_for_peer_via, both branches.

So a message whose first attempt went into a direct link that had already died (a half-open stream accepts frames until its keepalive notices) was never carried. By the time its acknowledgement timed out the link was gone, every resend was refused and went back on the retry queue, and a neighbour that could reach the recipient was never asked, for the whole 7-day outbox lifetime.

Found in the three-host lab run of #539: A sent to C over a direct peer stream, C was then reachable only through B, and minutes later A had 1 pending ACK, 1 queued retry and 0 mesh transmissions.

Change

offer_resend_to_mesh(message, accepted) in send.rs applies the first send's rule to every resend: offer when no carrier took the frame, and when one took it for a recipient can_reach_recipient says it cannot reach. It is called from the four resend branches. A resend over a link that still reaches the recipient is not offered, so an ordinary retransmission costs one copy.

The whole class, path by path:

Path	Before	After
First send refused / swallowed	offered	unchanged
Retry queue, refused	re-queued only	offered
Retry queue, accepted for an unreachable recipient	counted as sent	offered
Outbox flush (flush_outbox_all, flush_outbox_for_peer_via), refused / swallowed	re-queued / counted as sent	offered
ACK timeout	re-queues into the retry queue	covered by the retry loop
Parked-DM probe	offered at park; probe sends ran through the retry loop without an offer	offered at park and on each probe the relay or gateway does not reach

Why re-offering on every resend is safe (same argument as the park path): a neighbour that took the frame refuses further copies of the id for the RelaySeenCache retention window (600 s); the retry backoff bounds how often we ask; each offer spends own-send tokens like any other frame. Excluding neighbours already handed the id would be wrong: a neighbour records an id only when it *accepts* the frame, and one that refused it (rate limit, full queue) is the one a later offer should reach.

Not handled: a resend into a link that is half-open and not yet detected

While the carrier still lists the recipient as connected, the resend is accepted and can_reach_recipient is true, so nothing is offered. Offering there would not help anyway: offer_to_mesh hands a frame for a recipient who is a neighbour straight to that recipient, over the same suspect link. Routing around a link we still believe is up needs a "suspect link" state, which this PR does not add. The window is bounded by the carrier's own dead-link detection (the Python peer stream sets TCP keepalive 15 s + 3 x 5 s and TCP_USER_TIMEOUT 30 s on Linux), after which resends are refused and this fix carries them.

Tests

Five new cases in crates/offline-protocol/tests/mesh_forwarding.rs:

• a_message_lost_on_a_dead_direct_link_is_carried_by_the_mesh_on_retry: the first frame is swallowed by the direct link, the link dies, the retry goes through B, and A gets message_delivered. Red on main (nothing reaches C).
• a_flushed_message_no_carrier_takes_is_handed_to_the_neighbors
• a_resend_a_carrier_swallows_is_still_handed_to_the_neighbors
• a_flush_a_carrier_swallows_is_still_handed_to_the_neighbors
• a_resend_over_a_live_link_is_not_also_handed_to_the_mesh (cost side: exactly one resend copy, none to B)

Mutations, each turning a test red: removing any one of the four call sites, never offering an accepted resend, and always offering an accepted resend (dropping the reachability check).

Locally: cargo test --workspace --lib (engine lib 2015 tests), cargo test -p offline-protocol (all binaries), cargo clippy --workspace -- -D warnings, cargo fmt --all -- --check, RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps, all clean. No UDL change, no binding regeneration. Core and sealed untouched.

Related

• #539: examples/offline-first/run.py stops C before taking it off the shared network, so A sees the stream close. Once this lands that workaround can go: a half-open link now recovers through B on the next retry after the carrier notices.
• #537 fixes a separate settle bug, where a message that only the mesh carried never got message_delivered. The first test here does not depend on it: the first attempt was accepted, so an ACK was registered and the mesh-carried answer settles it on main.

Docs: docs/state-machines/outbox-and-retries.md, docs/mesh.md (mixed-neighbourhood section), docs/local-api.md, examples/offline-first/, CHANGELOG.md under Unreleased / Fixed.

Rebased onto main now that #537 has merged (a resend the mesh carries settles through its settle_dm_without_pending_ack). The last commit updates what main still said about the gap: docs/local-api.md and the offline-first demo's README and run.py now say a lost message crosses the mesh *late* (on the first retry after keepalive ends the dead stream), not never; the demo keeps stopping C first so the message lands inside its receive window.

Fixes #541.

Based on main (#538 and #537 merged in).

What

An offline-first demo on three devices, and the image changes it needs.

• examples/offline-first/compose.yml: three services from bindings/python/docker on two bridge networks (a - net-ab - b - net-bc - c), so b is the only way between a and c. No HTTP front, static peers (the topology is the file's, not multicast DNS's), one identity per volume.
• examples/offline-first/run.py (standard library only): runs the scenarios with offline-protocol-verify inside each device (docker exec, or ssh with the operator switching devices off and on), prints a table, exits non-zero on a failure.

1. store and forward: B stopped, A sends, B started: B receives, A's receipt within 60 s of B's return;

2. the same with A SIGKILLed and restarted between the send and B's return;

4. A and C pair over a temporary docker network connect, then share no network: C receives at hop_count 1 within 30 s, B emits message_relayed, and A gets C's receipt back across the hop. A missing receipt fails the scenario; --allow-missing-hop-receipt is for an image built from a 0.28.0 or earlier wheel, whose engine never sends it.

• examples/offline-first/README.md: the runbook, including the hardware legs (LAN between hosts, the carrier changing under a ping, relay, gateway daemon, phone) and a "What has been run" table.
• Image: OP_GATEWAY -> --gateway (+ two entrypoint test cases), config-ble.json and config-relay.json baked beside the default (one field different each), and the build now also fails when the installed package has no offline-protocol-verify.

What ran

In Docker 28.3 on one arm64 machine, run.py --fresh then run.py again, on an image whose native library was built from main after #537 (overlaid with this branch's Python sources): scenarios 1, 2 and 4 passed both times, scenario 4 with A's receipt required and back at hop 1 (receipts in 33 to 36 ms over wifiDirect in 1 and 2). An earlier pair of runs on the 0.28.0 native library passed 1, 2 and 4 without the receipt, as that engine predicts. The image was never built from the checkout. Nothing has run between separate hosts, over Bluetooth LE, a relay, a gateway daemon, or with a phone; the README says so.

scripts/tests/test-docker-entrypoint.sh passes (13 cases) and both scripts are shellcheck-clean; CI runs the test under /bin/sh on Ubuntu (dash).

Found while running it (engine, not fixed here)

The first lab run failed scenario 4 and the cause is real: a message that a direct stream took and then lost never crosses the mesh. I took C off net-ab before restarting it, so A kept a half-open stream to C for ~30 s; A's send went down it (pending ACK registered), and after keepalive ended the stream every retry was refused by the direct carriers and re-queued (protocol/mod.rs, the retry loop's Err arm) without offer_to_mesh, which only the first send's paths call. State minutes later on A: pending_ack_count 1, retry_queue_size 1, mesh transmissions 0. run.py now stops C while it is still on net-ab so A sees the close (the realistic case is the other one, and the README and the guide list it as a known gap). #537 does not cover it (it never touches the retry loop): filed as #541.

Also fixed on #538 while running this: state/pair connected under the verifier's application id and so took the message the recipient's service was holding for it; they now connect as <app id>.observer.

What

offline-protocol-verify, a command for the device a service runs on, and scenario tests that run the networking properties end to end through the local API.

The engine already reports every networking fact as an event: message_deferred while a recipient is away, message_relayed on a device that carries a frame for someone else, message_received with hop_count and transport, and message_delivered, the recipient's own acknowledgement naming the carrier it arrived over. The verifier sends through the service and waits for those events, so a check passes or fails on what the engine says.

Commands: state, send, await ID --until delivered|received, pair, ping, watch. JSON lines on stdout, a summary on stderr, exit 0 pass / 2 timeout / 1 terminal failure. await matches events by the id they carry, never by the server's correlation, so it works on a sender restarted since the send (and does not depend on the message_undeliverable terminal-tag fix).

The verifier has its own small client (verify/client.py) rather than sharing the HTTP front's persistent one: a command is short-lived, and the verifier must not depend on the front's package. Only watch reconnects, around it.

Scenario tests (tests/scenarios/)

Loopback peer streams, file stores with a store key, encryption on and required, as in the container image. A device is switched off (server and engine stop, stores released) and on again over the same stores and port.

• Store and forward: B off, A sends, message_deferred; B on, B gets the message, A gets message_delivered{transport: wifiDirect}. Also through the commands (await started while B is off).
• Reboot: B off, A sends, A restarts, B on, the restarted A delivers it (and its router never saw the id). Control: the same restart over an empty protocol-state directory delivers nothing.
• Through the middle: A and C meet once directly (the key exchange and Welcome never cross a hop), then C hears only B; A sends to C, C receives at hop_count == 1, B emits message_relayed.

Mutation checks, each red, then restored: dropping A's restart (reboot), dropping the pre-pair (C never receives: first contact does not cross a hop), keeping A and C linked (hop_count 0), keeping B on (no message_deferred).

Engine defect found: the sender's receipt never crosses the mesh

In the through-the-middle scenario C's acknowledgement is carried back by B (message_relayed from C to A) and reaches A, which drops it. A message whose only route is the mesh is refused by every carrier of the sender (handle_send_failure), so no pending acknowledgement is registered, and an acknowledgement with no pending record settles only a relay-parked DM (settle_parked_dm_from_ack). A keeps re-offering, C keeps re-acknowledging, and the outbox reports the delivered message failed after 7 days. The Rust simulator's the_answer_finds_its_way_back asserts only that B transmits toward A.

This is the bug #537 fixes. test_the_receipt_comes_back_through_the_middle_device records it as a strict expected failure: whichever of this PR and #537 merges second will see it XPASS and fail, and the marker comes off then. The guide and CHANGELOG say so.

Not covered

• A receipt the engine emits while no client of the application is connected is not held (only received messages are), so await/watch on the sender must start before the recipient can answer. Documented.
• Scenario tests skip on Windows (Unix socket); the TCP carrier has its own test.

Verified

• Python suite: 998 passed + 1 xfailed locally (macOS, 3.14).
• cargo test -p offline-protocol-uniffi --lib: 205 passed (the guard that walks every .py in the package).

Second commit (53cfad24)

Found by running #539 in containers: state and pair connected under the verifier's application id, so on the recipient they took the message the service was holding for that application and await --until received then timed out. They now connect as <app id>.observer; a test pins it (red with the old id). state no longer reports get_topology, which is empty on a running service. The guide also records a second engine gap seen there: a message that a direct stream took and then lost is retried over direct carriers only and never handed to the mesh.

Problem

With encryption required, a direct message to a peer the sender has never been next to cannot leave until a session exists. When the only path runs through a third device, the session never formed:

• The key package (send_key_package_to) and the Welcome (try_send_welcome) called transport_manager.send() directly and were never offered to the mesh. Every frame after them is sealed.
• The first push of a key package belongs to neighbour discovery, and a peer two hops away is never discovered: every frame from it arrives attributed to the device in the middle.

So the message sat in the pending queue for its whole seven-day lifetime.

Writing the test turned up a second defect on the same path. A message that no carrier took directly is handed to the mesh but registers no pending acknowledgement. The arm that settles an acknowledgement with nothing to match required a live park counter, which only a relay verdict creates. Result: a message to a peer two hops away (encryption on or off) arrived and was read, then stayed in the sender's outbox retrying until its lifetime failed it. message_delivered never fired.

Change

Commit 1: fix(protocol): a message the mesh carried settles when its recipient answers

• settle_parked_dm_from_ack becomes settle_dm_without_pending_ack. It now settles any plain DM still in the outbox when the recipient answers, whether or not it was parked.
• The rule that the answer must come from the recipient is unchanged and still applies to both settle paths.
• Rationale in the doc comment: the dropped park-counter gate only narrowed forgery to parties who could already deny delivery or forge an acknowledgement for an in-flight message.
• test_a_parked_dm_is_not_settled_by_anyone_elses_acknowledgement keeps its forgery half. Its second half flips: the recipient's own answer now settles the message.

Commit 2: feat(protocol): a session forms across the mesh between devices that never met

• Sender: rearm_key_package_for_peer, called from the reconciliation tick for peers with a queued message, now makes the first push itself when a peer has never been pushed to. It does so only when all of these hold:
• no carrier addresses the peer directly (can_reach_recipient)
• no infrastructure carrier claims it
• a mesh neighbour exists
• no session exists

Once a neighbour takes the push, the existing re-push ladder handles repeats (30 s doubling to 10 min). A push nobody took is retried on the next tick with the same pooled package (ADR 0012), so it mints no new key material.

• send_key_package_to and try_send_welcome: when no carrier takes the frame directly, hand it to offer_to_mesh. A carried Welcome takes the same in-flight path as one a carrier accepted, with the mesh confirmation timeout. The peer's probe or decrypt still settles it.
• Recipient: a key package from a sender only the mesh reaches is not answered with our own package first. The Welcome is all that sender needs. Sending a package back would make both sides build a group for the tiebreak to discard, with every frame crossing the middle device. The fresh package a new session sends for group invites still goes, after the Welcome.
• No new prefix, no UDL change, no binding regeneration.

Threat model

New residual R24: a session can be started from anywhere the mesh reaches. Anyone a mesh path connects to a device can make it build a group, store a package, and send a Welcome plus a package back across the mesh, once per freshly minted address. What bounds it:

• The package signature must derive to the claimed address.
• Every device in between applies its per-neighbour allowance and its per-second budget.
• The recipient's replies come out of the own-traffic share of its budget.
• Stored packages are capped at 1000, the push pool at 64, and a Welcome lifecycle at 300 s.
• There is no package reply first, so a package buys one Welcome, not a second group.
• A device with allowRelay off carries none of it.

Tests

• New protocol/tests/mesh_session.rs, built on real MLS over mock radios: alice and carol each hear only bob, and encryption is required.
• a_session_forms_across_the_mesh_between_devices_that_never_met:
• the message arrives once
• alice gets message_delivered with hop_count == 1
• alice's key package reached carol through bob, and carol's Welcome reached alice through bob
• carol's first handshake frame back is the Welcome, not a package
• bob carried nothing readable and surfaced nothing
• a_device_that_declines_to_carry_starts_no_session_between_others: with bob's allow_relay off, nothing of alice's crosses.
• New test_a_dm_the_mesh_carried_without_a_park_is_settled_by_its_answer, for commit 1.
• Mutation checks (each turned the named test red, then the code was restored):

Mutation	Test that went red
Drop the tick push	session test
Drop the key package mesh offer	session test
Drop the Welcome mesh offer	session test
Drop the reply gate	Welcome-first assertion
Restore the park-counter gate	both settle tests

• Local runs:
• cargo test --workspace: all green (engine lib 2018 tests)
• cargo clippy --workspace -- -D warnings: clean
• cargo fmt --all -- --check: clean
• RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps: clean
• core and sealed are untouched, so the no_std targets are unaffected.

Docs

docs/mesh.md (new "Starting a session across the mesh"), docs/state-machines/session-lifecycle.md, docs/state-machines/outbox-and-retries.md, docs/message-delivery.md, threat model R24, CHANGELOG.md (Added + Fixed).

Not covered

No device run. The simulator is mock BLE with lock-step rounds. In it the rate budgets refill on wall-clock time, so the session test raises burst/peer_burst. With the default budgets and no wall-clock time between rounds, the test's handshake stalls on rate_deferred (observed: 351 deferrals). How long the default budgets take in real time has not been measured.

Problem

The Python peripheral read its subscribed centrals from peripheral_manager_delegate._central_subscriptions, which exists only in bless's CoreBluetooth backend. On BlueZ (bless 0.3.0, the newest release) the read raised, peer tracking was silently off, every inbound fragment was attributed to the placeholder ble-peer, resolve_sender_identity returned at zero centrals, and the core had no BLE route back to a phone that connected to the box. Separately, the central and the peripheral both drained the core's one outbound fragment queue at once; the queue has no peek and no requeue (ble_return_fragment is a no-op), so a message's fragments could leave on two links and never reassemble.

Change

• Who the centrals are, on BlueZ (ble_peripheral.py, _BlueZCentrals). bless's BlueZ characteristic drops the options BlueZ passes to ReadValue/WriteValue, which name the calling device. The peripheral adds a dbus_next message handler on bless's own bus that reads options["device"] from calls to its own application's characteristics before bless dispatches them; it never answers or alters a call. A device becomes a central once it has read or written our characteristics, and stops being one when Device1.Connected is false or the object is gone (polled at the existing 1 s monitor interval).
• Telling a central from a peripheral our own central connected to: both are connected Device1 objects; only a central of ours calls into our application. That is the rule.
• Limits: a central that subscribes and never reads or writes is not seen (ours always read Device id and Identity first). BlueZ reports subscriptions per characteristic, not per device, so centrals are announced only while the Message characteristic has a subscriber, and with several connected the subscribed one cannot be named. Each inbound write is now attributed to its exact caller, so two centrals no longer collapse into ble-peer. Notifications still go to every subscribed central: neither bless backend addresses one.
• macOS is unchanged.
• One drain for both roles (protocol_manager.py, _BleTransportCallbackImpl). With both roles present one drain pops each fragment once and gives it to the central's link to its recipient (BleManager.write_fragment), else to the peripheral when a central is subscribed, else drops and counts it (fragments_dropped); the core's retry of the message re-sends it. The peripheral alone no longer pops while nobody is subscribed.
• docs/bridges/python.md P14, CHANGELOG under Fixed.

Verification

• 994 Python tests pass locally (macOS, Python 3.14); 15 new tests in test_ble_peripheral.py with a fake bus.
• 11 mutations, each turning its test red (attribution, path filter, subscription gate, drain gate, stale caller, disconnect, registration, Message-characteristic check, router dispatch, central-first order, drop counting).
• offline-protocol-uniffi lib tests (205) pass, including the guards that read the Python sources.

Not done: no board run

Nothing here has run on Linux hardware. Owed on a board:

1. bluetoothctl show: SupportedInstances, ActiveInstances, BlueZ version; whether another advertiser is active.

2. Start the service with ble_enabled: true; read the peripheral's start result and confirm the message handler sees the phone's ReadValue (the device path in the log).

3. A phone running the React Native example finds the box; messages deliver in both directions; neighbor_discovered.peer_id is the off1 address on both sides.

4. Phone disconnect is reported lost within a monitor tick or two.

5. With the phone bonded and unbonded, disconnect it and confirm the peripheral's drain stops taking fragments within a tick (BlueZ sends StopNotify for an unpaired central only; the gate now also requires a connected central).

6. Confirm the phone's StartNotify precedes its first write, so neighbor_discovered lands before the first inbound message and resolve_sender_identity has a central to bind.

7. Record hardware, BlueZ version and results in the runbook.

Problem

local_api/mux.py listed message_undeliverable in TERMINAL_TAGS, so the server forgot a message id the first time the engine reported the recipient away. The engine documents that event as non-terminal and repeatable (crates/offline-protocol/src/events.rs, MessageUndeliverable, lines 957-970): the message is parked, re-driven on reachability edges, and settles only with message_delivered or, at outbox expiry, message_failed. After the first report, every later event for that message, its delivery receipt included, was broadcast to every connected application. Both shipped example clients also stopped waiting at message_undeliverable, treating a recipient that is away as the outcome.

This is the path a store-and-forward run over the relay takes on every message: the relay answers recipient_unreachable, the sender parks, the recipient comes back, the sender gets message_delivered.

Change

• mux.py: message_undeliverable leaves TERMINAL_TAGS, with a comment saying why. Every other terminal tag is unchanged.
• bindings/python/examples/local_api_client.py and examples/local-api/client.mjs: wait for message_delivered or message_failed, print each message_undeliverable as a status line ({"undeliverable": ...}) and keep waiting; both take --deliver-timeout SECONDS (default 30, as before). Both now subscribe to message_failed.
• docs/spec/local-api.md: the correlated-routing rule says which event forgets which identifier and that message_undeliverable is not one. The event table row is unchanged (its routing column was already message_id). docs/local-api.md sending paragraph corrected. CHANGELOG under Fixed.

Tests

• test_an_undeliverable_message_stays_correlated_until_it_settles (router): two message_undeliverable then message_delivered all reach the issuing application only, and the id is forgotten after the delivery.
• test_the_examples_wait_past_undeliverable_for_the_delivery[python|node]: each example runs as a subprocess against a real server sending to an address that is away; the test injects two message_undeliverable and a message_delivered through the server's router, and asserts the output is hello, undeliverable, undeliverable, delivered, document, one_shot and that a client of another application heard none of it.
• Negative control: with the tag restored in TERMINAL_TAGS all three fail (the router keeps no id; the other application receives the broadcast receipt).
• Python suite: 982 passed locally (macOS, Python 3.14). cargo test -p offline-protocol-uniffi --lib local_api (the chapter guard) passes.

Not covered

• The undeliverable reports in the example test are injected, not produced by a relay verdict; no relay runs in CI.
• The HTTP front still answers 502 on the first message_undeliverable for a request (http_front/front.py:584). That is a choice for a request with a deadline, left as is; it now also sees the later events for that message instead of having them broadcast.

What

bindings/python/docker builds one service per host, on python:3.12-slim (Debian, glibc 2.41), with the lan and http extras.

• Dockerfile installs from PyPI, or a wheel built from a checkout dropped in wheels/. The local wheel is installed by itself first, so a PyPI release with the same version never replaces it. The build fails unless the installed service has the HTTP front and both extras import. The build context is that directory alone.
• entrypoint.sh maps OP_* variables to the service's flags, including OP_HTTP_TOKEN_FILE. It refuses to start without OFFLINE_PROTOCOL_STORE_KEY, with an OP_LAN other than 1 or 0, or with front options and no front.
• config.json enables the peer stream, with encryption required.
• compose.yml runs the service with host networking and a persistent volume.
• README.md lists what a host's container manifest must grant, in generic terms, and the failure each grant prevents:
• host networking, for multicast DNS and so the front sits on the host's loopback
• inbound TCP 7878
• a persistent volume, so the address survives a recreate
• the store key from a secret store, not beside the data
• the system D-Bus socket, for Bluetooth LE
• a clock

examples/http-front holds two programs:

• provider.py: standard library only, no SDK import. It answers /now with the time and the caller's authenticated address, and registers itself with its front. It retries while the front is unreachable or answers a 5xx, and stops at once on a 4xx with the front's reason.
• client.sh: a curl client.
• Both send the front's token when given its file (--token-file, OP_TOKEN_FILE).

tests/http_front/test_demo.py runs the provider as a separate process between two in-process encrypted hosts, and against a stub front that refuses (401, final) or is still connecting (503, retried). It skips outside a checkout.

Verified with Docker

1. Built the committed Dockerfile from a copy of the directory. It installed 0.28.0 from PyPI, and the native library loaded on glibc 2.41.

2. Overlaid this branch's Python sources in a throwaway layer. That is exact: nothing in the crates, the UDL or the generated bindings changed after v0.28.0.

3. Ran two containers on one bridge network with no OP_PEERS:

• They found each other over multicast DNS and announced each other under their proved addresses.
• A request from A, through A's front, to the demo provider in B returned 200 in 35 ms, with caller set to A's address.
• An unregistered service returned 404 unknown_service.

4. Restarted B. Its address and its front's registrations survived, and A was announced again.

The test containers, volumes and images were removed afterwards.

After the rebase onto main and two review rounds, also with Docker:

• With an empty wheels/, the build fails with "the installed offline-protocol-service has no HTTP front", because PyPI's 0.28.0 has no http extra. Before, it built and then crash-looped on unrecognized arguments: --lan --http.
• A 0.28.0 wheel with this branch's Python sources overlaid, in wheels/, is installed instead of PyPI's 0.28.0, and the front imports.
• In a running container: OP_LAN=yes exits 64. With OP_HTTP_TOKEN_FILE set, a provider without the token stops with the front's 401, and one given --token-file registers.

pytest bindings/python/tests -> 979 passed
shellcheck entrypoint.sh client.sh -> clean

Each new provider test turns red when its behaviour is reverted.

Not done

• The image cannot be built from PyPI yet. 0.28.0 has no http extra, so the build fails with that message. Until the next release, build with a wheel in wheels/.
• Bluetooth LE from a container has not run on hardware. The open question is whether BlueZ admits our advertisement beside another advertiser on the same adapter.
• Not run under host networking on a Linux host. The two-container run used a bridge network on Docker Desktop.

> Stacked on #529 and #530. This branch contains both of them, so the diff against main includes their commits until they merge. Review only the last commit, feat(bindings): the HTTP front. After either parent is squash-merged, merge main into this branch rather than rebasing it.

What

offline_protocol_sdk.http_front is the reference implementation of the chapter in #530. With it, an application that does not use the SDK can call a service on another device over plain HTTP:

# provider host: register an ordinary HTTP endpoint
curl -X PUT http://127.0.0.1:8080/services/timeofday -d '{"callback": "http://127.0.0.1:9000"}'
# requester host: call it by device address or operator alias
curl http://127.0.0.1:8080/now -H 'Host: timeofday.bob.offline.protocol.internal'

• offline-protocol-service --http HOST:PORT starts the front inside the service's own process.
• offline-protocol-http-front runs it as a separate process against a service that is already running.

How

• It is a client of the local API, not part of the server. It declares the application id offline-protocol-http-front. The server then stamps that id on its sends, routes its inbound messages to it, and holds them while it is away. Every rule of the local API bridge holds unchanged, and the server is untouched.
• Bodies travel only in sealed messages. Each request and response is a JSON envelope sent with send_message, which MLS encrypts. The front never uses the service request path, because those bodies are plaintext (R9).
• Only decrypted messages are acted on. The front ignores any message_received whose encrypted is not true. The callback receives the caller's MLS-authenticated address in X-Offline-Protocol-Sender.
• Responses are matched to the device that was asked. A response is accepted only if its id is pending and it comes from the device the request went to.
• Registrations are kept in a file. The front registers them with the engine again on every connection, because the engine and the server's ownership table forget them on a restart. A name another local application owns is answered with 409.
• Browse makes one discovery query per service per minute while any browser is open. Results stream as server-sent events, with the alias filled in when the operator's file has one.
• Failure events are matched by message id. message_failed or message_undeliverable for an outstanding request ends the wait early. An event that arrives before the send call has returned its id is kept until the id is known.
• Requests to this device's own address are served locally, without sending a message.
• The open questions from the plan were taken as recommended:
• aiohttp sits behind a new http extra, imported only when a front starts. It is added to dev and to the release audit list.
• No token is needed on loopback. Off loopback the token is required.
• Bodies are limited to 128 KiB. Deadlines default to 30 s, with a 300 s maximum and 60 s of clock slack.

Tests

tests/http_front/ holds 72 tests:

• Literals. The limits, the carried-header list, the application id, the domain and the error tokens are pinned as literals (C5).
• Hostnames. The grammar, and alias files that could mislead.
• End to end. Two engines with encryption on, joined by a loopback peer stream, each with a local API server and a front, and a stub provider that knows nothing of the SDK. Cases:
• the round trip, including status, headers, body and sender
• alias resolution and the local short-cut
• unknown_service from the far side
• unknown_device and bad_host
• body_too_large
• deadline_exceeded against an address nobody holds
• callback_failed and response_too_large
• browse
• registrations surviving a front restart
• the 409 conflict, unregister, malformed registrations, health, and the token
• Command line. --http options for the service, and the standalone command.

Negative controls: removing the encrypted check fails the plaintext test, and removing the sender check fails the wrong-device test.

pytest bindings/python/tests                     -> 897 passed
cargo test -p offline-protocol-uniffi --lib -- react_native_diagnostic_levels_match_every_bridge local_api_tables_partition_the_definition -> 2 passed

Not done

• Nothing has been run on two physical hosts yet.
• Request and response bodies are buffered whole, not streamed. 128 KiB is the ceiling.
• content-encoding is not a carried header. A compressed request body therefore reaches the callback without its header, so requesters should not compress.

What

This adds a new chapter, docs/spec/http-front.md. It describes how an application that does not use the SDK calls a service on another device over plain HTTP.

1. An application sends an ordinary request to <service>.<device>.<domain>.

2. The front on that host carries the request inside an end-to-end encrypted direct message.

3. The front on the other device calls the HTTP callback the providing application registered.

4. The answer comes back as an ordinary HTTP response.

The reference implementation follows in a separate PR, stacked on this one and on #529.

The decisions the chapter fixes

• Bodies travel only in sealed messages. They never use the service request path, because its bodies are signed plaintext (R9). On a LAN with no hop encryption, that path would expose every body to every forwarder.
• A front acts only on a message the engine reports as encrypted. The sender a provider is told is then the MLS-authenticated one. The application id stamped on the message routes the message to the front and authorizes nothing (R17).
• A device label is the 44-character address itself, or an operator alias. An address is a valid DNS label as it stands. An alias is never learned from a LAN record or a discovery response, because that would let any host bind a friendly name to its own authentic address.
• Limits:
• A raw body is at most 128 KiB. That keeps base64 plus the envelope under the engine's 256 KiB content cap.
• Headers are at most 8 KiB and the path at most 2 KiB.
• The deadline defaults to 30 s and is at most 300 s, with 60 s of clock slack on the provider.
• Discovery stays on the service path, with one query per service per minute while a browser is open. Results are claims: they never become aliases.
• Authorization:
• The front is open on loopback by default.
• Off loopback it requires a per-launch token, written to a file with mode 0600.
• The provider authorizes callers by reading X-Offline-Protocol-Sender.
• The residual risk is recorded as R23.

Also

• A row in the spec index.
• R23 in the threat model.
• P13 in the Python bridge rules.

Verified before writing

Two local API servers with encryption on exchanged a send_message over a loopback peer stream. The receiver's message_received carried encrypted: true, the sender's application id, and the MLS-bound sender, after 23 ms. That is the path the chapter relies on.

Docs only. No code, no UDL change.

What

offline-protocol-service now starts every transport its configuration enables, and the command line can switch on LAN discovery and name a relay.

Before this, the server started only the peer stream and the gateway client:

• ble_enabled built both Bluetooth LE managers and never ran either, so a host never advertised itself to phones.
• internet_enabled had no way to name a relay.
• The peer stream's DNS-SD advertising and discovery existed in the library and could not be switched on from the command.
• --listen and --peer were silently ignored when the configuration lacked wifi_direct_enabled.

Changes

• Start rule. After the engine has an address, the server starts the peer stream, the Bluetooth LE central and peripheral where the platform has a backend, the internet relay once configured, and the gateway once configured. A transport that fails to start is logged at error and stays stopped while the others run. The server fails only when none starts. The failure this prevents: an adapter another process holds, or a refused advertisement, taking the LAN path down.
• Flags. --lan switches on DNS-SD advertise and discover (it needs the lan extra and refuses at start without it). --relay URL configures the internet transport, with the token read from OFFLINE_PROTOCOL_RELAY_TOKEN or the variable --relay-token-env names.
• Refusals. --listen, --peer and --lan without wifi_direct_enabled, and --relay without internet_enabled, stop the command with the field named.

No server-synthesised event reports a failed transport. The local API chapter promises that events are the engine's JSON unchanged, so the failure goes to the log.

Tests

tests/local_api/test_local_api_carriers.py:

• both Bluetooth roles start with the server
• a refused advertiser leaves the peer stream running
• the server fails when no transport starts
• an unconfigured relay is not started
• each flag's configuration and refusal

The three server tests fail against the previous server.py, which serves as a negative control.

pytest bindings/python/tests  ->  825 passed

Not covered: a real Bluetooth adapter. Bluetooth is replaced by stubs in these tests. The first run on a Linux board with BlueZ is still owed.

Docs

• docs/local-api.md gains a "Which transports run" section.
• The Python README's service section is updated.
• CHANGELOG.md gains an [Unreleased] section.