Lesson 32: Fault Tolerance in Distributed Systems

Lesson 31 spread rooms over two nodes. More machines mean more things that fail: a node crashes, a cable is pulled, a switch reboots. Supervisors restart processes, but no supervisor restarts a data centre. We’ll look at what each failure does to the chat server and fix what copes badly.

Crash or Partition

From inside a node, both failures look the same. chat_cluster gets {nodedown, Node}. The other node may be dead, or running and thinking the same about us. No message can tell the two apart, because the network that would carry it is what failed. Code that reacts to nodedown must be correct in both cases.

How fast it comes depends on the failure. A VM that exits closes its TCP connections and the peer knows within milliseconds. A machine that loses power closes nothing. Connected nodes exchange ticks, and one that stays silent for net_ticktime seconds is declared down. The default is 60.

(a@localhost)1> net_kernel:get_net_ticktime().
60

Lower it in config/vm.args with -kernel net_ticktime 20, the same value on every node. A short tick time finds dead machines sooner, but disconnects a healthy node that stalls under load.

What a Dead Node Takes With It

Alice is on node a, Bob on node b. We stop a with halt(). and look from b.

(b@localhost)7> pg:get_members(chat, {room, <<"lobby">>}).
[<0.176.0>,<7543.213.0>]
=WARNING REPORT==== 19-Sep-2026::10:58:22.750850 ===
cluster: a@localhost is down, connected to []
(b@localhost)8> pg:get_members(chat, {room, <<"lobby">>}).
[<0.176.0>]

pg dropped the remote room, so publish/2 stops forwarding to it, global removed Alice’s name, and chat_cluster retries every five seconds. The server side needs no new code. What is left is Alice: her socket is closed, and her session token was in the memory of chat_auth on the dead node.

A Client That Comes Back

Behind a load balancer, a new connection to the same address reaches a live node. The client must retry, but not too eagerly: a thousand clients reconnecting in the same second can take down the survivor. Each failed attempt doubles the wait up to a ceiling, and a random part, the jitter, spreads clients that were cut off together.

Update priv/index.html. Replace the line that creates ws with two variables.

      let ws;
      let attempt = 0;

Remove the ws.onopen, ws.onclose and ws.onmessage assignments and add this below the handle function.

      function backoff(n) {
        const ceiling = Math.min(30000, 1000 * 2 ** n);
        return ceiling / 2 + Math.random() * ceiling / 2;
      }

      function connect() {
        ws = new WebSocket(`ws://${location.host}/ws`);
        ws.onopen = () => {
          attempt = 0;
          const token = sessionStorage.getItem("token");
          if (token) send({type: "auth", token});
        };
        ws.onclose = () => {
          const delay = backoff(attempt++);
          show("system", `connection lost, retrying in ${Math.round(delay / 1000)} s`);
          setTimeout(connect, delay);
        };
        ws.onmessage = (event) => handle(JSON.parse(event.data));
      }
      connect();

The auth message from lesson 21 logs the user in again. It only works if the node that answers knows the token.

Sessions on Every Node

Asking the other nodes with erpc fails in the one case we care about, because the node that issued the token is the one that died. The session must already be on the survivor. Mnesia replicates tables and chat_store already uses it, so sessions go into a ram_copies table on every node. That costs no disk write, and they survive as long as one node is up.

A node picks its role with a new primary key next to cluster_nodes, and both config files carry {primary, 'a@localhost'}. Two nodes on one machine also need separate data directories, so add {mnesia, [{dir, "mnesia_a"}]} to config/a.config and mnesia_b to the other.

Update src/chat_store.erl:

%% In src/chat_store.erl
-record(chat_session, {token, username, expires}).

-define(TABLES, [chat_user, chat_message, chat_session]).

%% The primary creates the tables. Every other node copies them.
init() ->
    case application:get_env(chat_server, primary, node()) of
        Primary when Primary =:= node() -> create_tables();
        Primary -> join_cluster([Primary])
    end.

join_cluster(Nodes) ->
    {ok, _Connected} = mnesia:change_config(extra_db_nodes, Nodes),
    mnesia:change_table_copy_type(schema, node(), disc_copies),
    ok = copy(chat_user, disc_copies),
    ok = copy(chat_message, disc_copies),
    ok = copy(chat_session, ram_copies),
    ok = mnesia:wait_for_tables(?TABLES, 10000).

copy(Table, Type) ->
    case mnesia:add_table_copy(Table, node(), Type) of
        {atomic, ok} -> ok;
        {aborted, {already_exists, Table, _Node}} -> ok
    end.

save_session(Token, Username, Expires) ->
    mnesia:dirty_write(#chat_session{token = Token, username = Username, expires = Expires}).

find_session(Token) ->
    case mnesia:dirty_read(chat_session, Token) of
        [#chat_session{username = Username, expires = Expires}] -> {ok, {Username, Expires}};
        [] -> {error, not_found}
    end.

delete_session(Token) ->
    mnesia:dirty_delete(chat_session, Token).

create_tables/0 is the body of init/0 from lesson 28, plus an ensure_table/2 call for chat_session with {ram_copies, [node()]}, waiting for ?TABLES. Export join_cluster/1 and the three session functions. A token is a fresh random key, so dirty operations are enough.

extra_db_nodes connects this node’s Mnesia to a running one and fetches the schema, change_table_copy_type/3 makes the local schema copy persistent, and add_table_copy/3 adds a replica of each table. On later starts the copies exist, which copy/2 accepts. A node that created its own tables earlier cannot join, so delete its Mnesia directory first.

In chat_auth the sessions map goes away and init/1 returns {ok, #{}}. The login clause calls chat_store:save_session(Token, Username, Expires) and logout calls chat_store:delete_session(Token). This is the verify clause.

%% In src/chat_auth.erl
handle_call({verify, Token}, _From, State) ->
    Now = erlang:system_time(second),
    case chat_store:find_session(Token) of
        {ok, {Username, Expires}} when Expires > Now ->
            {reply, {ok, Username}, State};
        {ok, _Expired} ->
            ok = chat_store:delete_session(Token),
            {reply, {error, expired}, State};
        {error, not_found} ->
            {reply, {error, invalid_token}, State}
    end;

We ran this with scripted clients: Alice logged in on a, we halted it, and a new connection to b sent her token.

(b@localhost)9> mnesia:system_info(running_db_nodes).
[b@localhost]
(b@localhost)12> ws_test_client:send_json(C, #{type => auth, token => Token}).
ok
(b@localhost)13> ws_test_client:flush_json(C).
[#{<<"token">> => <<"260B54524AC8735BC440...">>,
   <<"type">> => <<"login_ok">>,<<"username">> => <<"alice">>},
 #{<<"history">> => [],<<"room">> => <<"lobby">>,<<"type">> => <<"joined">>},
 #{<<"room">> => <<"lobby">>,<<"text">> => <<"alice joined">>,
   <<"type">> => <<"system">>}]

Accounts are replicated too, so Bob can register on b and log in on a. We also ran the browser client against a and restarted it. The waits below are real, and afterwards a loaded the session table from b with the token still valid.

3.0s connection lost, retrying in 1 s
3.9s connection lost, retrying in 1 s
5.3s connection lost, retrying in 4 s
8.9s connection lost, retrying in 6 s
15.3s [lobby] alice joined

Writing While a Replica Is Down

A Mnesia write, dirty or in a transaction, goes to every running replica, and one that is down does not block it. While a was down, Alice wrote a is down through b. When a came back it loaded its tables from b.

(a@localhost)1> mnesia:system_info(running_db_nodes).
[b@localhost,a@localhost]
(a@localhost)2> chat_store:recent_messages(<<"lobby">>, 10).
[#{type => message,text => <<"a is down">>,room => <<"lobby">>,
   from => <<"alice">>}]

This is safe because only one side changed, and Mnesia knows b saw a go down and never the reverse. The rule has a cost: a node that stopped first waits for the one that ran last. Started alone, b got no tables within ten seconds and init/0 failed until a was up.

When Both Sides Keep Writing

In a partition both nodes stay up, both see nodedown, and both accept writes. When the network heals, each side holds data the other lacks. Mnesia does not merge. It keeps both sides as they are, stops replicating between them, and reports {inconsistent_database, running_partitioned_network, Node} to subscribers of its system events. The application picks the winner.

Our policy is short. The primary’s data wins, and every other node reloads from it. Messages and sessions written on the losing side during the split are lost. For chat that is acceptable, because a few lost lines hurt less than a database that never agrees again. For payments it would be wrong, and you would need a store that refuses writes on the minority side.

Create src/chat_partition.erl:

-module(chat_partition).
-behaviour(gen_server).

-export([start_link/0]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2]).

start_link() ->
    gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).

init([]) ->
    {ok, _Node} = mnesia:subscribe(system),
    {ok, #{}}.

handle_call(_Request, _From, State) ->
    {reply, ok, State}.

handle_cast(_Msg, State) ->
    {noreply, State}.

handle_info({mnesia_system_event, {inconsistent_database, Context, Node}}, State) ->
    logger:error("mnesia: ~p with ~s", [Context, Node]),
    Primary = application:get_env(chat_server, primary, node()),
    case node() of
        Primary -> ok;
        _Other -> reload_from(Primary)
    end,
    {noreply, State};
handle_info(_Event, State) ->
    {noreply, State}.

%% Throw away our copy and load every table from the primary.
reload_from(Primary) ->
    logger:warning("mnesia: reloading all tables from ~s", [Primary]),
    stopped = mnesia:stop(),
    ok = mnesia:set_master_nodes([Primary]),
    ok = mnesia:start(),
    ok = mnesia:wait_for_tables(mnesia:system_info(local_tables), 30000),
    ok = mnesia:set_master_nodes([]),
    {ok, _Node} = mnesia:subscribe(system),
    ok.

Add it to the children of chat_server_sup with #{id => chat_partition, start => {chat_partition, start_link, []}}.

set_master_nodes/1 tells Mnesia where to load tables from at startup, whatever its own logs say. We clear it after the reload, because in our test a node that kept it could not load its tables when it later started alone. Stopping Mnesia ends the subscription, so we subscribe again. Calls into chat_store fail while it is down, and clients whose connection crashes reconnect.

A real partition is hard to stage on one machine. erlang:disconnect_node/1 alone does not work, because the nodes reconnect within milliseconds, but a wrong cookie keeps them apart. On a we ran this and saved a message:

(a@localhost)3> erlang:set_cookie('b@localhost', wrong), erlang:disconnect_node('b@localhost').
true
(a@localhost)4> chat_store:save_message(<<"lobby">>, <<"alice">>, <<"written on a">>).
ok

On b we saved written on b. Then erlang:set_cookie('b@localhost', chat) on a healed the split, and chat_cluster reconnected. This is the log of b.

cluster: a@localhost is up, connected to [a@localhost]
mnesia: running_partitioned_network with a@localhost
mnesia: reloading all tables from a@localhost

(b@localhost)20> [maps:get(text, M) || M <- chat_store:recent_messages(<<"lobby">>, 10)].
[<<"a is down">>,<<"written on a">>]

written on b is gone, as the policy says. Every transcript here is from a real run with two nodes on one machine. Detecting a machine without power is the one case we did not stage.

Exercises

Exercise 1: Rejoin rooms after a reconnect

After auth the server puts the user into lobby only. Make the client remember its rooms and rejoin them.

const rooms = new Set();
// add in "joined", delete in "left", and after "login_ok":
// rooms.forEach((room) => send({type: "join", room}));

Exercise 2: Tell users about a lost node

Give chat_cluster a job in its nodedown clause: broadcast a system event to the local users.

% Expected on every client of node b after node a stops:
% {"type":"system","text":"a@localhost is unreachable"}

Exercise 3: Refuse to be a minority

With three nodes, a side that sees fewer than half the expected nodes could stop accepting messages instead of losing them later.

has_quorum() ->
    Up = length(chat_cluster:connected()) + 1,
    All = length(chat_cluster:expected()) + 1,
    % true when more than half of all nodes are up
    Up * 2 > All.

% Expected with 3 nodes and 1 reachable peer: true
% Expected with 3 nodes and no peer: false

Key Takeaways

• nodedown does not say why. A crash and a partition look the same, so handlers must be right for both cases
• net_ticktime bounds how long a silent machine goes unnoticed. Lower values detect faster and disconnect busy nodes by mistake
• pg and global clean up after a dead node. State you keep yourself is your job
• Clients reconnect with exponential backoff and jitter, or the surviving node gets every client at once
• State needed after a failover must already be on the survivor, so sessions live in a replicated ram_copies table
• extra_db_nodes, change_table_copy_type/3 and add_table_copy/3 join a node to a Mnesia cluster. Writes skip replicas that are down, and those catch up on restart
• Mnesia detects a healed partition but never merges. Subscribe with mnesia:subscribe(system) and apply a policy you chose on purpose

Interactive Koans