What happens when something breaks

Kōlea is written in Elixir, which runs on the BEAM, the Erlang virtual machine. Ericsson built Erlang in the 1980s to run telephone switches, which were not allowed to go down, and that thinking is baked into everything it does.

Every web request, every background job, and every theme render runs in its own lightweight process. They share no memory. When one crashes, it crashes alone, and a supervisor starts a fresh one in its place. Erlang folks call this "let it crash", which sounds reckless until you compare it with one bad request taking down the whole server.

Your site's theme is a good example. Each render runs in its own process with a cap on how much memory it may use. A template that runs away with itself is killed at the cap and the rest of your site carries on.

Kōlea ships as one release that runs in one of two roles. Web servers answer requests and hand work off. Worker servers do the work: sending, importing, syncing with Stripe. A traffic spike on your site doesn't slow your broadcast down, and a big broadcast doesn't slow your site down.

The work itself is split into queues. Sign-in codes have a queue of their own, because someone is sitting on the code page waiting, and a broadcast to 100,000 readers shouldn't make them wait. Readers signing in to your site get a separate queue from creators signing in to Kōlea.

Background work runs on Oban, which keeps every job as a row in Postgres. Nothing is held in memory somewhere, so a restart, a deploy, or a server that falls over loses nothing. The job is still in the table when things come back.

A job is written in the same transaction as the change that caused it. When you send a broadcast, the mailing and the job that builds its recipient list commit together or not at all, so there is never a broadcast that exists with nobody going to send it. Outbound webhooks work the same way: if the change rolls back, nothing goes out.

If a worker dies in the middle of a job, Oban notices the orphan and puts it back in line. Jobs that fail retry with exponential backoff. Scheduled work, like the sequence clock every minute and the send reconciliation every five, runs on one elected node so two servers never do the same sweep.

Building a recipient list for a million readers happens in set-based batches, and each batch commits where it got to. If the server goes away halfway through, the next one picks up from there.

The send ledger is a table with one row per reader per mailing, and Postgres refuses a second one. A send job only names rows, and each row's own status decides what happens to it. Every status change moves a row only if it is still where the job expected to find it, so if Oban runs a job twice, or two workers grab the same work, the second one finds nothing to claim.

The case that keeps email people up at night is a timeout. Kōlea hands Mailgun a batch of 1,000 readers, the connection drops, and we have no idea whether Mailgun took it. Retrying could send 1,000 people the same email twice. Giving up could mean they never get it.

Kōlea does neither. Those rows are held, untouched, and once 30 minutes have passed a sweep asks Mailgun for its own record of that batch. Whatever Mailgun accepted is marked sent, and whatever it didn't goes back in the queue. The servers won't even boot if someone sets that wait shorter than ten times the longest a send call can take.

Every error from the mail provider gets sorted. When Mailgun asks us to slow down, we retry with backoff and it doesn't count as a failed attempt. Network errors and server errors retry until their attempts run out. A flat refusal, like an unverified From address, fails right away with the reason written on the row where you can see it.

There's a kill switch for the whole platform and one for each account. When one is on, mail goes into an outbox instead of out the door. Held mail is never thrown away: when the switch goes off there's a button to send it, each email is claimed before it goes so it goes once, and consent is checked again first. Sign-in codes are the exception, because a code from an hour ago is no use to anyone.

Consent is checked immediately before every call to the mail provider. If someone unsubscribes while your broadcast is halfway out, the rest of the send skips them.

When we deploy, each server raises a flag on its way down. Every send job finishes the email it's on and hands the rest back to the queue for another server to pick up.

Every table with your data on it carries your account's id, and Postgres row-level security makes every query say which account it's for. A query that forgets to say gets nothing back, which is the failure I want if there has to be one.

The rules that matter live in the database as constraints: required fields, checks, unique keys, and foreign keys. If the application code has a bug, the database still says no.

No process holds anything that belongs in a row. The feeder that hands out sending work is a process, and if it dies nothing is lost, because it wasn't holding anything. Every worker runs one and a Postgres lock picks exactly one to feed each window. It splits the sending budget by fair share, so someone mailing 500 readers isn't stuck behind someone mailing 500,000.

Mailgun and Stripe sign what they send us, and we check the signature before reading a word. If our database is having a bad moment we answer with an error and Mailgun tries again later. Stripe events are stored once per account, so a retried event is recorded once.

Webhooks you set up are signed with your endpoint's own secret, sent only to public https addresses (checked again before every attempt), retried with backoff, and logged attempt by attempt. An endpoint that keeps failing is switched off with the reason, so you're not left guessing.