How Lupyd Orders Concurrent MLS Commits: Epoch Races with mls-rs

In MLS, every commit moves a group from epoch N to epoch N+1. If two members both commit from epoch N, only one of them can become epoch N+1. RFC 9420 leaves the choice to the application. In Lupyd, the server decides: the first commit it validates wins, the other is rejected with an epoch error, and the losing client drops its pending commit and catches up. This post walks through the code that does that, in our open-source server and client, and the failure cases we know about.

Practical write-ups on this are rare. XMTP documents its epoch handling, and Cloudflare described a designated-committer design in its Orange Me2eets post. Smaller projects keep running into the same problem (for example this issue and this mls-rs fork PR). Everything below describes code in Firefly, our MLS and Signal messaging engine: the firefly-mls server and the firefly client, both built on mls-rs 0.54.

Why Concurrent Commits Collide

An MLS group's state is a chain of epochs. A commit is built against one specific epoch: it carries that epoch number, and its key schedule and tree changes only make sense on top of that state. RFC 9420 Section 14 (Sequencing of State Changes) says each commit is premised on a given starting state, so if two members send commits for the same epoch, only one of them can be applied, and the members need some way to agree on which. The protocol does not say how. RFC 9750, the MLS architecture document, leaves it to the deployment: in a strongly consistent design (Section 5.2.1), the Delivery Service is trusted to break ties when two members commit at the same time.

In a group chat this happens more than you might expect. One member adds someone while another renames a channel. A device renews its credential while an admin removes an old device. Background maintenance commits run on several devices at once. Every one of these is a commit, and they all race. For the broader picture of why groups need this machinery at all, see why group chat security is different.

Where Ordering Happens in Lupyd

The Lupyd server is an MLS external participant. It runs an mls-rs ExternalClient, keeps an ExternalSnapshot of every group in Postgres, and processes every commit the same way a member would, without being able to read application messages. For that to work, clients send commits and proposals as MLS PublicMessage (signed, not encrypted), and external commits are switched off:

// firefly: crates/core/src/rules.rs (FireflyMlsRules)
fn commit_options(&self, _new_roster: &Roster, _new_context: &GroupContext,
    _proposals: &ProposalBundle) -> Result<CommitOptions, Self::Error> {
    Ok(CommitOptions::new()
        .with_single_welcome_message(true)
        .with_allow_external_commit(false))
}

// Commits and proposals go out as PublicMessage, so the server can verify them.
fn encryption_options(&self, _current_roster: &Roster,
    _current_context: &GroupContext) -> Result<EncryptionOptions, Self::Error> {
    Ok(EncryptionOptions::new(
        false,
        mls_rs::client_builder::PaddingMode::None,
    ))
}

Because the server tracks the group's epoch itself, it can act as the single point that orders commits. Application messages stay end-to-end encrypted as PrivateMessage; the server only sees their epoch and group id. The trade-off of public commits is that the server sees group structure changes (who was added or removed, and the group's channel and role extension). We cover what the server can and cannot see in the Firefly overview.

How the Server Picks a Winner

Commits arrive over HTTP at POST /group/commit (or /group/commitAndWelcome when the commit adds members). Both handlers follow the same shape:

// firefly-mls: crates/server/src/group/mod.rs (trimmed)
pub async fn add_group_commit(sd: Arc<SharedData>, group_id: u64, by: &str,
    commit_message: &[u8]) -> Result<(u64, u32), InternalError> {
    let mut db = sd.db.get().await?;
    let tx = db.transaction().await?;

    let current_state = get_group_state(tx.client(), &sd, group_id).await?;
    let mut state_guard = current_state.write().await;   // one writer per group

    let (commit_message_id, epoch, group, committer_address) =
        add_commit(&sd, tx.client(), group_id, by, commit_message, &state_guard).await?;

    tx.commit().await?;
    *state_guard = group.snapshot().to_bytes()?;        // advance only after COMMIT
    // ... re-insert into group_state_cache, then fan out to members
    tokio::spawn(notify_group_members(sd, message, group, committer_address));
    return Ok((commit_message_id, epoch));
}

The ordering comes from state_guard. Each group's current snapshot lives in a tokio::sync::RwLock, and accepting a commit holds that lock for writing from the moment the server loads the state until after Postgres has committed. Two commits for the same group cannot be processed at the same time in one server process. Whichever request takes the lock first is the winner. There is no tie-break on member index, role or timestamp; it is first come, first served.

Inside add_commit, the server loads the snapshot into an mls-rs ExternalGroup and calls process_incoming_message. That call does the full MLS validation: signature, membership, the proposals against FireflyMlsRules, and the epoch. The epoch check is the part that decides the race:

// mls-rs 0.54.0: src/group/message_processor.rs
match content_type {
    ContentType::Commit => {
        if context.epoch != epoch {
            Err(MlsError::InvalidEpoch)
        } else {
            Ok(())
        }
    }
    // ...
}?;

// mls-rs 0.54.0: src/client.rs
#[cfg_attr(feature = "std", error("plaintext message for incorrect epoch"))]
InvalidEpoch,

When the winner's commit is processed, the group advances to N+1. The loser's commit still says epoch N, so when it gets the lock, mls-rs returns InvalidEpoch. On the HTTP path, InternalError::Mls is mapped to 400 Bad Request with the error's display text as the body, which is plaintext message for incorrect epoch. Nothing is written, and the transaction is rolled back when it is dropped.

If the commit is valid, add_commit also checks group limits (channels, roles, member count) and rejects with 403 if a commit would grow the group past them. Then it writes three things in the same transaction: the new snapshot into groups.state, the commit into group_commits, and a copy into group_messages. Only after tx.commit() succeeds is the in-memory state replaced and the commit pushed to other members.

The Database Backstop

The in-memory lock is per process. The schema adds a second guarantee that does not depend on it:

-- firefly-mls: initdb.sql
CREATE TABLE IF NOT EXISTS group_commits (
    id BIGINT NOT NULL DEFAULT now_us(),
    group_id BIGINT NOT NULL REFERENCES groups (id) ON DELETE CASCADE,
    epoch INT NOT NULL,
    msg BYTEA NOT NULL,

    PRIMARY KEY (group_id, epoch)
);

CREATE TABLE IF NOT EXISTS group_messages (
    id BIGINT NOT NULL DEFAULT now_us(),
    group_id BIGINT NOT NULL REFERENCES groups (id) ON DELETE CASCADE,
    msg BYTEA NOT NULL,
    epoch INT NOT NULL DEFAULT 0,
    PRIMARY KEY (group_id, id)
);

group_commits is keyed on (group_id, epoch), where epoch is the epoch the commit produces. Postgres will hold at most one commit per epoch per group. If two transactions ever got as far as inserting a commit for the same epoch, the second insert would fail on the primary key and its transaction would roll back.

The state itself is loaded through a small cache:

// firefly-mls: crates/server/src/group/mod.rs (trimmed)
pub async fn get_group_state(db: &impl GenericClient, sd: &SharedData, group_id: u64)
    -> Result<Arc<RwLock<Vec<u8>>>, InternalError> {
    if let Some(value) = sd.group_state_cache.lock().unwrap().get(&group_id) {
        return Ok(value.clone());
    }
    let original_state = Arc::new(RwLock::new(Vec::new()));
    {
        let mut writing_state_to = original_state.write().await;
        let q = "SELECT state FROM groups WHERE id = $1 FOR UPDATE";
        let rows = db.query_typed(q, &[(&(group_id as i64), Type::INT8)]).await?;
        // ... copy the stored ExternalSnapshot bytes into writing_state_to
    }
    sd.group_state_cache.lock().unwrap().insert(group_id, original_state.clone());
    Ok(original_state)
}

The cache is a SieveCache of up to 1000 groups, each entry an Arc around the lock. On a miss, the state is read with SELECT ... FOR UPDATE inside the commit's transaction, so the groups row stays locked until that transaction ends. On a hit, no row lock is taken up front; the UPDATE groups SET state inside add_commit still locks the row for the rest of the transaction.

The copy in group_messages is stored with the epoch the commit was framed in (N), while group_commits stores the epoch it produces (N+1). The code comments explain why: in group_messages, epoch means "the epoch this was sent from" for every row, so a client that already applied the commit sees it as an old message and skips it.

What the Losing Client Does

On the client, mls-rs keeps a commit you have built as a pending commit until you either apply or clear it. Firefly only applies it once the server has accepted it:

// firefly: crates/core/src/lib.rs (trimmed)
match self.send_commit_to_server(commit).await {
    Ok(id) => {
        let before = roster_leaves(group);
        let desc = group.apply_pending_commit().await?;
        self.record_departures(&before, group);
        group.write_to_storage().await?;
        return Ok(id);
    }
    Err(err) => {
        log::info!("server rejected commit: {:?}", err);
        group.clear_pending_commit();
        return Err(anyhow::anyhow!(err));
    }
}

Any non-2xx response, including the 400 from a lost race, is an error. The client calls clear_pending_commit(), so its local group stays exactly at epoch N, and returns the error to its caller. There is no automatic rebuild-and-resend loop in the core library. What happens next depends on who asked for the commit:

• Background maintenance (renewing this device's credential in each group, removing gone devices) logs the failure with the comment "usually a concurrent commit by another member" and tries again on its next periodic check.
• User actions such as adding a member, changing roles or editing channels return the error to the app.

One detail matters here: the client holds its group's Mutex for the whole round trip, from building the commit to applying or clearing it. Incoming messages for that group, including the winner's commit, are processed through the same lock, so they wait until the loser has cleared its pending commit. The client never tries to process someone else's commit while it still has one of its own pending.

How the Loser Catches Up

After a successful commit the server fans it out to every member device except the committer's: over the live connection if the device is online, otherwise with a push notification so it syncs. The pushed message carries the new epoch (N+1). Devices that were offline fetch missed commits from group_commits, where the query returns commits above the epoch the device last reported, ordered by epoch.

Whichever way the commit arrives, the client runs the same gate before handing it to mls-rs:

// firefly: crates/client/src/websocket.rs, on_group_message (trimmed)
let current_epoch = group.epoch().await;

if msg.epoch < current_epoch as u32 {
    log::info!("Skipping old message (group {} id {})", groupId, msg.id);
    return Ok(());
}

if is_commit && msg.epoch == current_epoch as u32 {
    log::info!("Skipping redundant commit message (group {} id {})", groupId, msg.id);
    return Ok(());
}

let message = group.process(msg.message.to_vec()).await?;

For the losing client at epoch N, the winner's commit arrives labelled N+1 (from a push or group_commits) or N (from group_messages). Neither is skipped, so it is processed and the client moves to N+1. Re-delivered commits (for example, commit bundles re-pushed on reconnect) are labelled with their resulting epoch, so one that equals the local epoch is skipped as redundant. If the user still wants the change they tried to make, it has to be rebuilt on top of N+1 and sent again.

Avoiding Races Before They Happen

First-come ordering is safe, but losing races wastes work, and some flows invite many members to commit at once. Re-adding a device that has fallen out of a group is one: every online member sees the request. Firefly coordinates it with a short lease in Postgres before anyone builds a commit:

-- firefly-mls: claim_re_add, crates/server/src/group/mod.rs
UPDATE group_re_add_requests
   SET claimed_by = $1, claimed_at = now_us()
 WHERE group_id = $2 AND address_id = $3
   AND (claimed_by = 0 OR claimed_by = $1 OR claimed_at < now_us() - 30000000)
   AND EXISTS (SELECT 1 FROM addresses WHERE id = $1 AND username = $4)
RETURNING 1

A member must claim the request through POST /group/reAdd/claim before committing. The claim succeeds if nobody holds it, the caller already holds it, or the existing claim is older than 30 seconds (now_us() is in microseconds). Everyone else gets 409 Conflict and stops. Non-owners also wait 1.5 seconds before trying to claim, which gives the group owner a head start. This does not replace the epoch check, it only reduces how often two members build the same commit.

Failure Cases

The lost-response gap

The last row deserves a closer look. The client only learns that its commit won from the HTTP response. If the server commits the transaction and the response is lost (a dropped connection, a timeout), the client takes the error branch and clears its pending commit, even though the group has moved to N+1 with its own commit.

The server does not push a commit back to the device that sent it, but that device will later receive it through sync. If the commit carried an update path (credential renewals and other path-carrying commits do), mls-rs cannot process it: decrypting a path from your own leaf needs the private state kept in the pending commit, and without it mls-rs returns CantProcessMessageFromSelf. The device is then stuck behind the group. Firefly has a re-add flow (POST /group/reAdd, then another member claims and re-adds the device) that can bring such a device back into the group, but we have not built automatic detection of this specific case, and the window is small. We would rather document it than pretend it does not exist.

Frequently Asked Questions

Final Takeaway

Commit ordering in MLS is not hard to get right if one component owns it. In Lupyd, that is the server: a per-group lock gives first-come ordering, mls-rs's epoch check rejects anything built on an old state, and a primary key in Postgres makes sure only one commit per epoch is stored. Clients only apply a commit after the server accepts it, and drop it otherwise. The weak spot is a response lost after the server has committed, which we have described above. If you are comparing MLS with Signal-style groups, libsignal vs MLS covers the bigger picture.

Download Lupyd to use MLS-encrypted group channels, or read the code at github.com/lupyd.

Last updated: 3 November 2026. Code excerpts are trimmed from the firefly-mls and firefly repositories and mls-rs 0.54.0 as of this date.