How we're rebuilding our email client around one Go engine

We have had an email client for many years, and it serves us well. But it is really several apps. The Android app is very different from the iOS app, and both are different from the desktop app. A bug fixed on Android can live on in the desktop app, and the other way around. Each app has its own email parser and its own way of handling the network, and keeping devices in sync is hard.

So I asked myself: with what recent LLMs can do, could we rebuild the stack around one Go codebase, a single engine that serves every app? The idea is not new. Tailscale does it, and Proton does something similar.

The target is macOS, Windows, Linux, iOS and Android. The UI is the small part of that. The hard part is everything behind it:

• syncing years of mail over IMAP without losing or duplicating a message,
• storing it so that a list of 50,000 threads scrolls without touching the network,
• parsing MIME written by every mail program since the 90s,
• threading, search, drafts, sending, encryption.

Written once per platform, that is five copies of the hardest code, each with bugs that show up on only one platform. So we’re writing it once, in Go, as a headless engine, and every app is a shell around it.

This post walks through how that fits together: the boundary between the engine and each UI, how each platform reaches it, what lives inside it, and how it copes with IMAP servers that do not behave the way the RFCs say they should.

The whole picture#

Three things to take from it:

1. Every host reaches the same interface. The desktop app, the mobile app and our developer CLI all end up calling the same Call and Subscribe.
2. Only the engine talks to mail servers. No shell opens a socket to an IMAP server, parses a message or writes to the database.
3. The UI never waits on the network. Every view renders from the local store. The network catches up in the background.

The engine’s surface is one function#

The whole engine, as a shell sees it:

type Engine interface {
	Call(ctx context.Context, method string, params []byte) ([]byte, error)
	Subscribe(fn func(event []byte)) (cancel func())
	Close() error
}

Call takes a method name and JSON, and returns JSON. Subscribe streams events, also as JSON: threads.changed, sync.state, body.ready and so on. That is the whole API.

The obvious design is one Go method per operation, ListThreads(folder, limit), MarkRead(ids), with a binding for each platform. That gets painful fast:

• Every bridge has to know every method. Adding mail.snooze means touching the desktop bridge, the mobile binding and the CLI.
• Mobile bindings are picky about types. gomobile (more on it later) exports only simple types: no maps, no slices of structs, no generics. Rich signatures need hand-written adapters.
• Versioning is per method. An older shell and a newer engine disagree in many small places instead of one.

With one function, a bridge is a byte pipe. It moves a method name and two JSON blobs and never decodes them. A new method touches the engine and the code that calls it, and no bridge.

Two more things come for free. A shell can ask engine.methods what this build supports instead of guessing, so a build without sync simply has no sync methods. And a product can add its own methods and events through a Module without forking the engine. Our key management and contacts are built that way.

type Module struct {
	Name    string      // namespace for methods, storage dir, backup section
	Methods Methods     // added to the surface; may not shadow a built-in
	Events  []EventShape
	Send    SendHook    // runs on every mail.send
	Start   func(*Core, Scope)
	Close   func() error
}

Types without stubs#

“Everything is JSON” sounds like “everything is untyped”. It isn’t, because we generate the types from Go. A generator reads the Go types the engine returns and writes a TypeScript file that maps each method name to its params and result:

const page = await engine.call('mail.threads', { limit: 50 })
//    ^? ThreadsResult, looked up from the method name

The generated file is committed, and CI regenerates it and fails if anything changed. The Go and TypeScript types cannot drift apart without someone noticing.

How each platform reaches the engine#

The engine package imports no transport. It doesn’t know whether it runs in a child process, inside an iOS app or in a test. Each platform has a small bridge that turns its native way of talking into Call.

Desktop: a sidecar process over stdio#

On desktop, the UI is a web page inside Electron. The engine is a separate binary that the shell starts. They talk JSON-RPC 2.0 over the child’s stdin and stdout, one JSON object per line.

Why a separate process and not a native Node addon:

• Crashes stay contained. A panic in the engine doesn’t take the window down. The shell restarts it.
• Upgrades are independent. A native addon has to be rebuilt for every Electron and Node version. A process just speaks lines of JSON.
• It runs on its own. echo '{"jsonrpc":"2.0","id":1,"method":"mail.threads"}' | engine works in a terminal, so we test the process handling without Electron.

Why stdio and not a localhost HTTP or WebSocket server:

• There is no port. Anything on the machine can connect to a localhost port. A pipe belongs to the parent process alone, so there is nothing to authenticate. A test makes sure the engine never opens a listening socket.
• Lifetime comes free. When the shell closes stdin, the engine finishes its running calls and exits. When the shell dies, the pipe closes and the engine follows.

On top of the pipe, the bridge adds a handshake (rpc.ready with a version and the method list), request ids so a call can be cancelled with rpc.cancel, and a cap on work in flight so a slow UI cannot stall the engine.

The UI only knows an interface with call and subscribe. Besides the Electron implementation, there is one that serves data recorded by the CLI, so the whole UI runs in a plain browser tab with no engine. That keeps UI work and screenshot tests fast.

Mobile: the same engine as a library#

iOS doesn’t let an app run a long-lived child process, and Android makes it awkward. So on mobile the engine is linked into the app. gomobile bind builds an .aar for Android and an .xcframework for iOS, both exposing the same Call and Subscribe. Because the surface is “a string and some bytes”, the binding stays tiny and gomobile’s type limits never come up.

The mobile differences show up as engine features, not as forks:

• Bytes are expensive across the bridge. A []byte crossing gomobile is copied, and our JSON encodes it as base64. So the engine never returns a message body or an attachment as bytes. It returns a file path, and the app reads the file with its own APIs.

• No IDLE in the background. IMAP can’t wake a phone, so the wake comes from a push sent from outside it, and the app gets a short window before it is suspended, about 30 seconds on iOS. The engine has a bounded entry point for that:

  sync/loop/delta.gogo Copy

  // DeltaOnce runs one bounded pass over an account, for a mobile wake,
  // and returns when it is done or the budget is spent.
  func (g *Group) DeltaOnce(ctx context.Context, 
                            account int64, 
                            budget time.Duration) error

  New mail comes first, and older history gets whatever time is left. If the budget runs out, the rest waits for the next wake. Stopping halfway loses nothing, because the sync position only moves forward in the same transaction as the messages it covers.

CLI and tests: no bridge at all#

Our developer CLI imports the engine and calls engine.Call directly. Its -json output is exactly what the desktop and mobile apps parse:

$ cli import -account you@example.com testdata/corpus/*.mbox.gz
$ cli call mail.threads '{"limit":5}'

When a bug report says “the thread list is wrong”, we reproduce it with the same engine method the app called, without the app.

Inside the engine#

Local-first#

Every read comes from the local store. Every user action is a local change first:

The outbox is a table in the same database, so a crash between steps 1 and 3 loses nothing. Offline is just a state where step 3 waits.

Storage: index, cache and files#

The store has three pieces, each with a different job:

The rule that keeps this manageable: blobs/ is the local source of truth for mail, and everything derived from it is a cache. When we fix the HTML sanitizer or change the search tokenizer, we delete content.db and rebuild it from the files on disk, fetching again only what was evicted.

The split also gives us a locked state. meta.db opens at launch, so the thread list and envelope search work right away. content.db opens only when the user unlocks. Until then, message bodies answer locked. This matters most for encrypted mail: its blob stays encrypted as received, and the decrypted text only ever lives in content.db.

Why files and not SQLite blobs? SQLite is faster for small blobs, but at mail scale it hurts: VACUUM on a 40 GB database needs exclusive access and twice the disk, a 100 MB attachment goes through the WAL, and one corrupt page can damage the whole store instead of one message. On mobile, a file path is also free to hand to the platform.

A message is one message row with one placement row per location. A Gmail message with five labels is one row, five placements and one file. Moving a message updates a placement and downloads nothing.

Sync: one actor per account#

IMAP connections are scarce and slow. A connection has one selected folder at a time, a command can’t be taken back once sent, and providers limit how many connections an account can open. Meanwhile three kinds of work compete for them:

• now: the user opened a message, clicked a folder, sent a mail,
• soon: new mail arrived, bodies near what is on screen,
• eventually: years of history, expunge sweeps.

The scheduler follows a few rules:

1. One actor per account. Accounts share no lock and no goroutine, so one broken account cannot stall another.
2. A small connection budget. At most four per account: one IDLE watch and three task slots.
3. Three classes. Interactive runs first. Background and Backfill split the rest 3 to 1.
4. No preemption, only small steps. IMAP cannot cancel a running command, so long work is cut into bounded steps that queue their own next step. A click waits at most one step.
5. Keyed work. Asking twice for the same folder’s resync merges into one task, and a merge can promote it to a higher class.

Boundaries enforced by tests#

Architecture diagrams go stale. We keep this one honest with a test that loads the package graph with go/ast and fails the build on a forbidden import or call:

{
	name:  "engine-transport-free",
	scope: exceptPkg(under("engine"), under("engine/rpc", "engine/mobile")),
	deny: anyImport(
		stdPkgs("net", "net/http", "net/rpc"),
		pathContains("go-imap", "go-smtp", "grpc", "websocket"),
	),
},

Other rules in the same file: the engine reaches sync only through a narrow port, modules don’t import each other, every IMAP command has a deadline, and nothing calls net.Listen. Each rule carries the reason it exists, so a failure explains itself.

One engine on many devices#

A user with a laptop, a phone and a work desktop runs three copies of the same engine against one mailbox, each with its own local store.

The engines never talk to each other. The server is the only shared state. Following the IMAP RFCs’ advice for disconnected clients, the local store is a cache of the server, and the only thing a device owns is the list of changes it has not sent yet. That shapes a lot of small decisions:

• Flag changes are deltas. The outbox sends +FLAGS (\Seen) or -FLAGS, never FLAGS (...), which would wipe flags another device just set.
• Incoming sync doesn’t overwrite pending local changes. Otherwise a message marked read on this device would flip back to unread until the outbox drains.
• The connection budget is shared. Providers limit connections per account, not per device. In one of our runs against Yahoo, three of eight concurrent logins came back NO [LIMIT] LOGIN Rate limit hit. So each engine keeps its own share small, staggers its logins, and backs off when the server says no.
• Sends are never blindly retried. A device that crashed mid-send can’t know whether the server took the message. It separates “never handed over” from “handed over, no answer”, retries only the first, and asks the user about the second.

IMAP: the protocol and what servers actually do#

IMAP is the hardest part of the engine. The RFCs describe a reasonable protocol. Real servers implement parts of it, advertise parts they don’t implement, and sometimes return a successful answer that is wrong.

So we stopped trusting documentation early. Every claim in our provider notes comes from a recorded session against a live server, and those recordings replay in our tests.

Servers disagree in three ways#

The standard way to sync cheaply is CONDSTORE: remember the highest modification sequence you have seen, and next time ask only for what changed since:

SELECT INBOX (CONDSTORE)
UID FETCH 1:* (FLAGS) (CHANGEDSINCE 812345)

On a 30K-message Gmail inbox, that returned about 100 changed messages in under half a second. Asking from the start took 12 seconds. Doing the cheap one every cycle is the whole game.

Yahoo advertises CONDSTORE and then refuses the fetch:

T8 NO [CANNOT] UID FETCH Command is not supported on this IMAP server

The second difference is how to list every UID in a folder, which is how you notice deletions. Gmail supports ESEARCH, so UID SEARCH ALL returns compact ranges: 3 KB for 30K messages, 400 times smaller than UID FETCH 1:* (UID). Yahoo has no ESEARCH, and there SEARCH moved fewer bytes but took 2 to 7 times longer than FETCH.

The third is whether the server shows you the whole folder at all. That one turned out to be dangerous.

A folder view can be silently truncated#

We filled a Yahoo folder with 12,500 messages and opened it through Yahoo’s two IMAP hosts at the same moment:

== default host ==                   == export host ==
T10 EXAMINE "Test"                   T10 EXAMINE "Test"
* 10000 EXISTS                       * 12500 EXISTS
* OK [UIDVALIDITY 1787899885]        * OK [UIDVALIDITY 1787899885]
* OK [UIDNEXT 14501]                 * OK [UIDNEXT 14501]

Same mailbox, two front-ends. The default host serves only the newest 10,000 messages. UID SEARCH ALL and UID FETCH 1:* both return the truncated set, with OK. The capability lists are identical, so nothing hints at the difference.

Think about what a naive client does with that. Deletion detection is “list the server’s UIDs, compare with the local ones, delete what’s missing”. Here that silently deletes the oldest 2,500 messages from the local store, on every sweep.

The only signal is that STATUS, asked before selecting the folder, still reports the true count. So every enumeration goes through a gate:

• The check is one-sided (EXISTS < STATUS), because new mail arriving between the two commands raises EXISTS.
• One mismatch is not enough, because another client expunging a message also lowers EXISTS. On a re-read, an expunge shows up in both counts and a truncation only in one.
• A truncated folder keeps syncing new mail and flags. It just stops deleting, and the account shows as degraded rather than broken.

This gate catches any server that serves a partial view, including ones we have never tested.

Three layers: generic, probed, overridden#

One algorithm can’t serve every server, and a table keyed on provider name goes stale. So the sync strategy for an account is built in three layers:

1. Generic. The base protocol (SEARCH, FETCH, STATUS), extensions like CONDSTORE where they actually work, and safety checks like the truncation gate.
2. Probed. At first connect, the engine tests what the server actually does instead of trusting what it advertises. For example, it sends one CHANGEDSINCE fetch whose correct answer it already knows. A few messages back means the server honors it, every message means it ignores it, and NO means it refuses. The result is stored per account and probed again when the capability list changes.
3. Overridden. Only what no probe can discover, such as “Yahoo’s export host serves the whole mailbox, so prefer it and fall back to the default host”. Each override names a host and must have a fallback.

The rule between layers: a lower layer may choose a mechanism, but may never relax a safety check. An override can say “connect elsewhere”. It cannot say “skip the completeness check”. Otherwise the check would protect only the providers someone already investigated.

A short list of other surprises#

Each of these came from a recorded session and now has a test that replays it:

• \Deleted is not a mark. On both Gmail and Yahoo, STORE +FLAGS (\Deleted) removed the messages on the spot. A client can’t treat setting that flag as reversible.
• Gmail’s expunge does not delete. Expunging from a label removes the label. The message stays in All Mail. Only moving it to Trash deletes it.
• OK is not proof, and neither is its absence. Gmail dropped a connection and still stored a message whose OK never arrived. Yahoo answered OK to 1,000 appends, then the next SELECT reported 1,000 messages followed by 0, with no EXPUNGE. Retrying based on what the client thinks it sent would have duplicated mail.
• UIDVALIDITY is weaker than it looks. On Yahoo it is the folder’s creation time in seconds, so two folders created in the same second share it, and it can’t tell folders apart on its own.
• “Folder gone” and “server error” look the same. Yahoo answers both with NO [TRYCREATE] SELECT error - Folder does not exist or server encountered an error. So a failed SELECT is never treated as a deletion.
• IDLE timeouts vary. The IMAP spec expects a server to keep an idle connection for at least 30 minutes. Yahoo hangs up after 5, so the engine renews IDLE every 4.
• Servers rewrite messages. Yahoo adds a References header pointing at a parent message that doesn’t exist, so threading has to cope with that.

The general lessons: decide from measured behavior, not from capability lists or provider names. Treat any count, OK or missing response as something to check. And keep every destructive decision (deleting local mail, resending) behind a check that holds even for a server nobody has looked at yet.

Why Go#

Go has held up well for the engine. What helped most:

• One codebase for every target. GOOS and GOARCH cover desktop, and gomobile covers iOS and Android. A sync bug fixed on desktop is fixed on the phone.
• Goroutines fit IMAP. An IMAP connection is a blocking conversation, one command at a time, so a goroutine per connection reads like the protocol. context.Context carries a cancel from the UI down through the engine.
• Testing is built in. CI runs the scheduler under -race, and the parsers and the bridge are fuzzed with the built-in fuzzer.
• The tooling is a library. go/ast and reflect made the boundary tests and the TypeScript generator cheap to build.
• One binary. The desktop sidecar is a single file with no runtime to install.

What it costs:

• cgo for encryption. SQLCipher is a C library, so every target needs a C toolchain.
• gomobile is narrow and copies bytes. That pushed us toward the one-function surface and file paths.
• Binary size and GC on phones. Not a problem so far, but worth watching.

What we would keep#

We started with several apps, each with its own parser, its own network code and its own bugs. The rebuild replaces that with one engine and a thin shell per platform. So far, these are the decisions we would make again on day one:

1. One headless engine, thin shells. The engine owns sync, storage, parsing and search. A shell renders and forwards input.
2. One function as the API. Call(method, json) plus an event stream, with types generated from Go.
3. Transport at the edge. The engine never knows if it runs over stdio, inside a phone app or in a test.
4. Local-first with an outbox. The UI reads the local store, and the server stays the source of truth.
5. Measure the servers. Probe what they actually do, and keep safety checks no override can switch off.
6. Enforce the architecture in tests. If a boundary matters, a test fails when someone crosses it.

The first app built on the engine is now in beta on macOS and Windows, with a handful of users. We’ll see how it holds up against their mailboxes.