Protobuf Everywhere: How Lupyd Uses Protocol Buffers Across Web, Mobile, and Databases

When developers hear Protocol Buffers, they usually assume gRPC comes with it. The two tools originated from Google and are frequently paired together, but they address entirely different problems. Protocol Buffers is a binary serialization format; gRPC is an RPC transport framework. Across Lupyd and its sub-projects, Firefly and Noon, we use Protobuf throughout our web, desktop, and mobile clients, client-facing HTTP endpoints, authentication tokens, database records, and cryptographic workflows, all without running gRPC on our client connections.

Separating Protobuf from gRPC

Protobuf does not require HTTP/2 streaming, special RPC stubs, or bespoke networking gateways. At its core, Protobuf takes structured data defined in a schema file and encodes it into compact binary bytes. What you do with those bytes is up to you:

• Send them over a standard HTTP/1.1 or HTTP/2 POST request using fetch() or curl.
• Stream them over WebSockets or WebRTC data channels.
• Store them directly on disk or in a database column.
• Feed them into cryptographic hashing and encryption routines.

Adopting gRPC means accepting its full networking stack: HTTP/2 framing, bidirectional streaming models, and specialized tooling like gRPC-Web proxies for browser support. For many architectures, especially web and mobile client applications, that infrastructure adds operational complexity without proportional benefit.

Decoupling the serialization format from the transport layer gives us the benefits of compact binary data and generated types while keeping our transport layer standard, stateless HTTP.

The Binary-First Architecture

Lupyd and its core systems, including Firefly and Noon, are built around a binary-first architecture. In our stack, binary is the native format for in-memory operations, network transmission, and persistent storage.

Text formats such as JSON are treated as auxiliary encodings. They exist exclusively at the outer boundaries of our systems when we need to provide backwards compatibility for third-party integrations, external webhooks, or public developer APIs. Within our own client-to-server and internal communication paths, data remains binary from start to finish.

Avoiding text-based serialization eliminates the recurring costs of string parsing:

• No Key Name Duplication: In JSON, field names like "conversation_id" and "sender_user_id" are repeated in every record. Protobuf replaces these with small numeric field tags (varints).
• Direct Memory Decoding: Binary decoding reads fixed-width bytes and varints directly into typed structures, bypassing heap allocations and string scanning.
• Compile-Time Contracts: Data structures are locked into strongly typed interfaces across every platform, eliminating field mismatches and silent serialization bugs.

Protobuf Web Tokens in Firefly

Most web services rely on JSON Web Tokens (JWT) for stateless authentication. A standard JWT consists of three separate JSON objects (header, payload, and signature) that are each base64url-encoded and joined by dots:

// Standard JWT: text serialized to JSON, then re-encoded to base64url strings
eyJhbGciOiJFUzI1NiJ9.eyJzdWIiOiJ1c3JfODkiLCJleHAiOjE3Mzg5... .MEQCIG3x...

Every time a client attaches a JWT to a request and every time a server verifies it, both sides must perform string splitting, base64 decoding passes, and JSON string parsing before they can even inspect the user's identity or verify the signature.

In Firefly, we replaced JWT with a custom Protobuf Web Token (PWT). A PWT defines the authentication envelope directly as a Protobuf message:

syntax = "proto3";

package firefly.auth.v1;

message ProtobufWebToken {
  bytes user_id = 1;
  int64 issued_at = 2;
  int64 expires_at = 3;
  uint32 key_version = 4;
  repeated string scopes = 5;
  bytes signature = 6;
}

With PWT, authentication operates purely on raw bytes. The user identifier is stored as compact bytes, the timestamps and key versions are encoded as varints, and the cryptographic signature (such as an Ed25519 signature) sits directly in a byte field.

There are no base64 string conversions, no string-concatenation routines, and no JSON parsing passes on auth checks. The token decodes in a single step directly into native memory, and the signature verifies immediately over the serialized claims buffer.

Form Schemas and Submissions in Noon

Lupyd's anonymous feedback and voting platform, Noon, relies on RSA Blind Signatures (RFC 9474) to decouple a participant's identity from their submission.

In Noon, both the form creation schema and the respondent's form submission are modeled in Protobuf. This choice is critical for the cryptographic guarantees of blind signatures.

During the Noon protocol, the respondent computes a cryptographic digest of their submission before blinding:

If SubmissionPayload were encoded as JSON, differences in client platforms (key ordering, comma spacing, or integer vs float representation) would produce different byte sequences for identical answers. That discrepancy would invalidate the unblinded signature when the user submits their response to the verification server.

Using Protobuf ensures that:

1. Form Definitions Are Structured: Questions, input types, validation constraints, and options are compiled into typed models across the Noon form builder and client renderer.
2. Submissions Are Byte-Deterministic: The respondent's answers serialize into an exact binary buffer regardless of whether the form is submitted from a browser, a desktop client, or a mobile device. The SHA-256 hash matches precisely, and the blind signature verifies without issue.
3. Wire-Level Answer Validation: When a response arrives, the Protobuf decoder validates that required fields and answer types match the form schema without needing separate validation libraries.

Unified Cross-Platform Code Generation

Lupyd maintains clients across multiple platforms and languages:

• Web Frontend: TypeScript and React.
• Desktop and Core Crypto: Rust and Tauri.
• Mobile: Kotlin on Android, Swift on iOS.
• Backend Services: Rust and Go.

Maintaining data models manually across four distinct languages is error-prone. A typo in a property name or a mismatched type definition in one client can quietly cause runtime issues.

With Protobuf, all data structures are defined once in version-controlled .proto files:

syntax = "proto3";

package lupyd.messaging.v1;

message ChatMessage {
  string message_id = 1;
  string conversation_id = 2;
  string sender_id = 3;
  bytes encrypted_body = 4;
  int64 timestamp_ms = 5;
  optional string reply_to_id = 6;
}

Running our build pipeline generates native types and serialization functions across all targets:

• TypeScript interfaces and binary encode/decode functions for web clients.
• Rust structs implementing prost::Message for desktop and Firefly.
• Kotlin data classes for Android.
• Go structs for backend API handlers.

When an API field changes or a new message type is added, updating the .proto file updates the models across every repository. Type mismatches become compile-time errors instead of production bugs.

Built-in Schema Validation Without Extra Libraries

In JSON-based APIs, verifying incoming data requires dedicated schema libraries such as Zod, Joi, or custom validator functions. Teams write and maintain validation logic to check:

• Whether required fields exist.
• Whether values match expected types (strings, numbers, booleans).
• Whether unknown fields are allowed or rejected.

This approach duplicates effort: backend engineers define validation in Go or Rust, while frontend engineers define identical schemas in TypeScript. If the two definitions fall out of sync, requests fail unexpectedly.

Protobuf handles schema validation directly during deserialization. The decoder reads binary wire tags, maps them to known field types, and constructs the object. If an incoming byte stream contains malformed data or type mismatches, the decode operation fails immediately.

This eliminates thousands of lines of handwritten validation boilerplate. The schema defined in the .proto file is the sole contract, enforced automatically at the binary decoding boundary.

Client HTTP Endpoints Over Plain HTTP

Instead of running gRPC-Web proxies or custom streaming protocols for client communication, Lupyd's API endpoints use standard HTTP POST requests with a Protobuf payload.

In the web client, sending a message uses the standard browser fetch() API:

// Encode client payload to binary bytes
const body = ChatMessage.encode({
  messageId: "msg_90412",
  conversationId: "conv_441",
  senderId: currentUserId,
  encryptedBody: ciphertext,
  timestampMs: BigInt(Date.now()),
}).finish();

// Standard HTTP POST request
const res = await fetch("/api/v1/messages/send", {
  method: "POST",
  headers: {
    "Content-Type": "application/x-protobuf",
    "Accept": "application/x-protobuf",
  },
  body: body,
});

// Decode binary response
const buffer = await res.arrayBuffer();
const ack = MessageAck.decode(new Uint8Array(buffer));

This architecture retains full compatibility with existing web infrastructure:

• Edge Networks & CDNs: Cloudflare and AWS CloudFront handle binary request and response bodies without special streaming configuration.
• Standard Middleware: Rate limiters, WAFs, authorization headers, and cookies function exactly as they do with JSON.
• Debugging Fallbacks: Endpoints can support content negotiation. If an external service or developer tool sends Accept: application/json, the backend can serialize the same internal model into JSON on demand, while keeping binary Protobuf as the default for Lupyd clients.

Storing Protobuf Wire Formats in the Database

Many modern web applications store semi-structured data or event logs inside relational databases using JSON columns (such as PostgreSQL's JSONB).

While flexible, JSON storage comes with operational costs:

• Storage Size: Repeating field names in every row consumes substantial disk space and reduces cache efficiency in database memory (shared buffers).
• Serialization Overhead: Every query requires the database or the application to parse text into memory structures and re-serialize them on writes.

In Lupyd, complex event objects, form records, and internal messages are stored directly as raw Protobuf wire format bytes (using binary columns such as PostgreSQL BYTEA or SQLite BLOB).

This provides three clear operational benefits:

1. Compact Storage: Binary wire format is significantly smaller than text JSON, reducing table size and improving buffer cache hit ratios.
2. Faster Reads: Decoding raw binary bytes in application memory takes microseconds, avoiding the string scanning and allocation overhead of JSON parsers. In cases where the server merely relays stored events to a client, it can stream the raw bytes directly to the HTTP response without deserializing them at all.
3. Safe Schema Evolution: Protobuf is explicitly designed for forward and backward compatibility. When we add a new optional field with a new tag number, older server binaries or background workers continue reading existing rows without errors or database migration scripts.

Deterministic Message Encoding in Firefly E2EE

Lupyd's end-to-end encryption engine, Firefly, relies on mathematical guarantees: digital signatures, cryptographic hashes, and MLS (Messaging Layer Security) epoch state transitions.

In cryptography, signatures and hashes must be computed over exact, unambiguous byte sequences. A single modified byte invalidates a signature.

JSON serialization is inherently non-deterministic across platforms and programming languages:

• Key ordering varies depending on language implementations: {"a":1,"b":2} versus {"b":2,"a":1}.
• Whitespace handling (spaces after colons or commas) differs across JSON libraries.
• Floating-point numbers can be formatted with or without trailing zeros.

If a client running JavaScript signs a JSON message and a recipient running Rust or Kotlin verifies that signature, minor discrepancies in key ordering or formatting will cause signature verification to fail.

Firefly eliminates this ambiguity by using Protobuf for pre-encryption serialization:

Performance and Resource Improvements

Across our client and server workloads, adopting binary Protobuf yields consistent performance improvements:

• Payload Size: Message payloads over the wire are typically 60% to 80% smaller compared to equivalent JSON representations, primarily due to the removal of field names and efficient varint integer encoding.
• CPU and Memory Overhead: Decoding binary bytes avoids extensive heap string allocations. In benchmarks, Protobuf deserialization is consistently multiple times faster than standard JSON parsing libraries.
• Mobile Efficiency: Smaller network payloads reduce cellular radio on-time, and faster decode routines reduce main-thread CPU spikes, resulting in measurable battery savings and smoother UI frame rates during offline sync events.

Real-World Limitations and Quirks

While Protobuf provides clear advantages, it comes with practical trade-offs that teams should consider before adopting it broadly:

1. No Native UUID Type

Protobuf does not include a dedicated UUID primitive. Applications that use UUIDs must choose between two approaches:

• Store as string: A 36-character string representation (e.g. "550e8400-e29b-41d4-a716-446655440000") is simple to inspect but consumes 36 bytes on the wire for 16 bytes of data.
• Store as 16 raw bytes: Using a bytes field is mathematically optimal but requires application-level conversions to and from hex strings whenever IDs are logged or displayed in UIs.

2. Dynamic Types are Unwieldy

Protobuf supports unstructured data via types like google.protobuf.Struct and Any. However, using these types requires manual packing, unpacking, and runtime type checking. If an application relies heavily on dynamic, arbitrary key-value dictionaries, Protobuf adds complexity without providing its core benefit: static compile-time safety.

3. Default Values and Field Presence in Proto3

In Proto3, fields originally defaulted to zero-values (empty string, 0, false) and were omitted from the wire when unset. This made it difficult to determine whether a field was intentionally set to 0 or simply not provided.

Later releases reintroduced the optional keyword to Proto3, restoring explicit field presence tracking while maintaining backward compatibility. Teams must still be deliberate about when to rely on default zero-values versus explicit optional presence checks.

Conclusion

Protocol Buffers is not tightly coupled to gRPC. It is a capable, compact serialization format that functions effectively over plain HTTP, inside databases, and across client applications.

By separating the data format from the transport framework, Lupyd, Firefly, and Noon maintain standard, stateless web infrastructure while operating on a unified binary foundation across every layer of the stack.