Comparing Signal and MLS Protocols for Modern Group Chat Security

Securing a private message sent between two people is one of the most mature achievements in modern applied cryptography. However, the moment a conversation expands from a two-person direct dialogue to a dynamic team workspace with dozens or hundreds of participants, the underlying mathematical and operational assumptions change completely.

To understand why digital communication systems are structured the way they are, developers and security-conscious teams must understand the foundational distinction between session-based pairwise encryption and tree-based group key agreement.

1. Introduction: The Complexity of Group vs Direct Messaging

In a direct conversation between two people, both participants maintain a private, synchronized cryptographic channel. When Alice sends a message to Bob, her device encrypts the payload directly for Bob's cryptographic session. If Bob replies, his device acknowledges Alice's ratchet state and advances the session. If one device is temporarily offline, the server simply queues the encrypted envelope until the recipient reconnects.

When moving to group conversations, this simple bilateral model experiences immediate friction:

• Dynamic Membership: Group rosters change continuously. Teammates join channels, contractors depart, and devices are replaced. Each membership change requires modifying cryptographic access immediately.
• Forward & Backward Secrecy Boundaries: When a member departs, they must not be able to read future messages (Forward Secrecy upon eviction). When a new member joins, they must not decrypt past history prior to their arrival (Backward Secrecy upon addition).
• Bandwidth & Computational Fanout: Attempting to manage a 100-person group by running 99 individual pairwise sessions per user requires quadratic $O(N^2)$ key renegotiations and heavy payload re-encryption on mobile hardware.

2. How libsignal Works in User-to-User Chats

The Signal protocol (implemented in libraries like libsignal) represents the gold standard for bilateral end-to-end encryption. It solves the challenge of establishing asynchronous, forward-secure communication between two devices without requiring both parties to be online simultaneously.

Key Building Blocks of Pairwise Sessions:

• Identity Keys & Prekeys (X3DH): When initializing a session, User A fetches User B's published public KeyBundle (comprising an Identity Key, Signed Prekey, and One-Time Prekeys) from the relay server. A performs an Extended Triple Diffie-Hellman exchange to derive a shared root key before transmitting the first message.
• The Symmetric Key Ratchet: For every message sent in a single direction, a Key Derivation Function (KDF) chain advances. Each message receives a unique, single-use encryption key. Even if an attacker captures a single message key, they cannot derive past or future message keys in that chain.
• The Asymmetric Diffie-Hellman Ratchet: Whenever the conversation alternates (User B replies to User A), both parties perform a new ephemeral Diffie-Hellman exchange. This injects fresh mathematical entropy into the root key, achieving Post-Compromise Security (Break-in Recovery), which restores conversation confidentiality even if device memory was temporarily inspected.

User A (Sender)
   │
   ▼
[ Local Key Derivation & Ratchet Step ]
   │
   ▼
[ Authenticated Ciphertext Payload ]
   │
   ▼
[ Blind Transport / Relay Server ] (Cannot read payload)
   │
   ▼
[ Authenticated Ciphertext Payload ]
   │
   ▼
[ Local Ratchet Step & Decryption ]
   │
   ▼
User B (Recipient)
  

Why Pairwise Sessions Excel in 1-to-1: In a two-party dialogue, pairwise ratcheting is lightweight, robust, and mathematically elegant. Because there are no other participants to synchronize, session state is contained entirely on the two communicating endpoints.

3. How Signal Handles Group Chats Today: Sender Keys & Private Groups

A common misconception in secure messaging discussions is that Signal simply re-encrypts every group message pairwise to every member. If that were true, sending a message in a 100-person group would require the sender's phone to run 99 separate Double Ratchet operations, and sending 50 messages would mean running nearly 5,000 distinct encryptions. That would quickly drain phone batteries and saturate mobile upload bandwidth.

To avoid this penalty, Signal designed the Sender Key protocol alongside its Private Group system. Here is how group messaging actually operates in Signal today:

While Sender Keys make everyday group messaging fast and lightweight, they introduce distinct cryptographic and operational trade-offs:

• The Member Leave Penalty: To maintain backward secrecy (ensuring a departed member cannot read future messages), whenever any participant leaves or is removed from the group, all remaining members must discard their Sender Keys. Every sender must generate a new key and re-distribute it over pairwise sessions to every remaining member. In a 500-member group, a single departure can trigger a storm of pairwise key exchanges.
• Weaker Post-Compromise Security (PCS): In 1-to-1 Signal chats, the Double Ratchet heals itself continuously whenever parties alternate messages, thanks to the Diffie-Hellman ratchet. In group Sender Keys, there is no asymmetric ratchet during normal messaging. If an attacker extracts Alice's active Sender Key from a compromised device, they can decrypt all future messages Alice posts to that group until Alice rotates her key (which only happens upon membership changes or after a time threshold).
• Private Group Metadata: Signal combines Sender Keys with Anonymous Credentials and zero-knowledge proofs. The server stores an encrypted group roster and validates membership access tokens using keyed-hash verifications, ensuring the server never learns who is in the group or what the group is named.

4. Signal's Active Role in the MLS Design and Implementation Plans

Because Signal developed the Double Ratchet and Sender Keys, some assume MLS was built as a competing standard by outside organizations. In reality, Signal researchers and cryptographers actively helped design and shape the IETF MLS standard (RFC 9420) from its inception.

Working alongside teams from Cisco, Mozilla, Wire, Oxford University, and Inria, Signal's team brought years of operational experience with real-world end-to-end encrypted messaging to the IETF working group. The limitations Signal encountered while scaling Sender Keys in production directly motivated the requirements for MLS:

• True Asymmetric Post-Compromise Security in Groups: MLS introduces TreeKEM, where members can update their cryptographic path in $O(\log N)$ operations, providing groups with the same self-healing post-compromise security that Signal pioneered for 1-to-1 direct messages.
• Logarithmic Group Membership Operations: Replacing the $O(N)$ pairwise key-distribution storm of Sender Keys with logarithmic tree commits, allowing groups to scale to thousands of participants smoothly.
• Unified Multi-Device State: Modeling every user device as a distinct leaf in the key tree, eliminating the need to fan out separate pairwise copies across multiple linked devices.

How Signal Plans to Adopt MLS

Signal has actively researched implementing MLS into its core architecture to replace or upgrade Sender Keys for group messaging. However, deploying MLS across a production network with hundreds of millions of users introduces concrete distributed-systems hurdles that Signal is methodically addressing:

5. How MLS Works for Group Communication (TreeKEM & Epochs)

While pairwise ratcheting works flawlessly for two participants, using it for groups of 50, 500, or 5,000 members forces the sender's client to encrypt and upload $N-1$ separate ciphertexts for every single message.

The IETF developed Messaging Layer Security (MLS / RFC 9420) to solve this fundamental scalability bottleneck. Instead of individual pairwise sessions, MLS establishes a single shared group cryptographic state synchronized across all members.

             Alice (Admin / Sender)
                       │
                       ▼
        ┌─────────────────────────────┐
        │  Shared Group Epoch Secret  │
        │     (MLS / TreeKEM Tree)    │
        └─────────────────────────────┘
             ▲        ▲        ▲
             │        │        │
           Bob      Carol    David
  

Core Concepts in MLS Group Security:

• Group Epochs: Group state is organized into sequential numerical epochs. Every message sent within a given epoch is encrypted using key material derived from that epoch's master secret.
• TreeKEM (Tree-Based Key Agreement): Group participants occupy leaf nodes in a balanced binary tree. Parent nodes hold derived key pairs. By updating only the nodes along their path from leaf to root, a member can update the group secret in logarithmic $O(log N)$ operations instead of linear $O(N)$ pairwise renegotiations.
• Proposals and Commits: Changes to group membership (adding a member, removing a member, or rotating keys) are proposed and then finalized through a signed Commit message. The commit ratchets the group into a new epoch ($K o K+1$).

6. libsignal vs MLS: Architectural Comparison

Neither protocol is universally "better" than the other; rather, each was engineered to solve a fundamentally different cryptographic coordination problem:

7. Why Lupyd Can Use Different Approaches for DMs and Groups

The decision to utilize distinct cryptographic mechanisms for direct dialogues and multi-member team channels is rooted in engineering practicality:

1. Zero-Overhead Direct Messages: In a private two-person chat, there are no group roster proposals, no tree leaf allocations, and no epoch commit races. A lightweight pairwise ratchet delivers maximum performance, instant delivery, and minimal device battery usage.
2. Scalable Team Governance: In team workspaces, channels frequently add new colleagues, manage channel permissions, and remove departing personnel. TreeKEM in MLS ensures that revoking a user's cryptographic access does not require re-encrypting separate messages for dozens of other team members.

By separating these two conversation modalities, the architecture avoids forcing a single protocol into an operational domain it was not originally optimized for.

8. Could MLS Be Used for Both 1-to-1 and Groups in the Future?

A recurring question among distributed systems engineers is whether modern secure messengers should eliminate pairwise protocols entirely and use MLS as a single unified cryptographic engine across all chat types.

Mathematically, a 1-to-1 conversation can simply be defined as an MLS group with exactly two members ($N=2$).

"The right choice depends on Lupyd's actual security requirements, implementation maturity, interoperability needs, performance requirements, and long-term architecture."

9. The Decisive Driver: Multi-Device Fanout in Firefly

Beyond theoretical protocol elegance or codebase consolidation, there is one overriding, pragmatic engineering reason why Lupyd is actively evaluating switching to MLS even for 1-to-1 direct messages: multi-device synchronization overhead.

In Lupyd's Firefly messaging engine, each user account supports up to 5 linked devices (such as an iPhone, an Android tablet, a work laptop, a home desktop, and a travel device). When users take advantage of all 5 device slots, the traditional pairwise Double Ratchet model introduces severe computational and network fanout, even in a simple one-on-one direct message.

Pairwise Double Ratchet (1 message = 9 encryptions + 9 uploads):
Alice Phone ──┬──→ [Encrypt Bob Phone]      ──→ Upload Payload 1
              ├──→ [Encrypt Bob Laptop]     ──→ Upload Payload 2
              ├──→ [Encrypt Bob Work PC]    ──→ Upload Payload 3
              ├──→ [Encrypt Bob Tablet]     ──→ Upload Payload 4
              ├──→ [Encrypt Bob Desktop]    ──→ Upload Payload 5
              ├──→ [Encrypt Alice Laptop]   ──→ Upload Payload 6 (Self-sync)
              ├──→ [Encrypt Alice Work PC]  ──→ Upload Payload 7 (Self-sync)
              ├──→ [Encrypt Alice Tablet]   ──→ Upload Payload 8 (Self-sync)
              └──→ [Encrypt Alice Desktop]  ──→ Upload Payload 9 (Self-sync)

Messaging Layer Security (1 message = 1 encryption + 1 broadcast):
Alice Phone ──→ [Single Epoch Encrypt] ──→ Upload Single Payload
                                                    │
                                                    ▼ (Blind Server Broadcast)
                     ┌──────────┬──────────┬────────┴─────────┬──────────┐
                     ▼          ▼          ▼                  ▼          ▼
                 Bob Dev 1  Bob Dev 2  Bob Dev 3 ...    Alice Dev 2  Alice Dev 3 ...
  

In the pairwise architecture, mobile endpoints bear a heavy, repeated penalty. If mobile network conditions are degraded or battery is low, computing 9 separate cryptographic ratchet steps and uploading 9 independent ciphertext blobs consumes noticeable CPU cycles, battery power, and upload bandwidth.

With MLS, the conceptual model flips: every device is simply a member of the group. The blind relay server receives one ciphertext and delivers it to the other 9 endpoints. Because each device holds a valid leaf in the conversation's TreeKEM tree, every device can independently decrypt the exact same payload. This eliminates the multi-device fanout penalty entirely, making MLS a vastly more scalable choice as user device counts grow.

10. What Would Need to Change in Lupyd? (Future Possibility)

If a messaging platform like Lupyd were to evaluate a transition toward a unified MLS architecture, several fundamental subsystems would undergo architectural consolidation:

Current Architecture Concept (Specialized per chat modality):

1-to-1 Direct Messages ───────→ Pairwise Sessions (libsignal / Signal-style)
Team & Group Channels  ───────→ Tree-Based Key Agreement (MLS / RFC 9420)


Possible Future Unified Architecture (Conceptual exploration):

1-to-1 Direct Messages ───────┐
                              ├───→ Unified MLS Engine (2-member & N-member groups)
Team & Group Channels  ───────┘
  

High-Level Components of a Unified Migration:

• Client State Machine Consolidation: Endpoint applications would operate a single MLS group state engine, initializing 1-to-1 chats as 2-leaf TreeKEM trees.
• KeyPackage & Identity Directory: User public keys would be published exclusively as standardized MLS KeyPackage bundles.
• Blind Commit Sequencing: The relay server's sequencing layer would handle commit ordering uniformly across both direct dialogues and team workspaces.

Note: This represents a theoretical architectural evaluation of protocol consolidation, rather than an active roadmap commitment.

11. Beyond Protocols: Holistic Security & Privacy Considerations

In secure systems engineering, a fundamental truth is often overlooked: a cryptographic protocol alone does not make an application secure. The security of any messaging platform depends equally on the surrounding systems architecture and client-side implementation: