Protocol Security Analysis - Signal Protocol Implementation

Overview

Comprehensive security analysis of X3DH (Extended Triple Diffie-Hellman) key agreement and Double Ratchet protocol implementation, including compliance with Signal Protocol specifications and security properties.

Analysis Date: January 2025 Implementation: Rust (src/rust/x3dh.rs, src/rust/double_ratchet.rs) Overall Protocol Rating: 🟡 MEDIUM (Functional but with specification deviations)

---

Executive Summary

The Signal Protocol implementation correctly implements the core cryptographic flows of X3DH and Double Ratchet, but contains specification deviations that may affect security and interoperability.

Key Findings:

• ✅ X3DH implements proper 4-DH handshake
• ✅ Double Ratchet provides forward secrecy and PCS
• ❌ CRITICAL: AAD not used in AES-GCM encryption
• ❌ CRITICAL: HKDF parameters reversed (salt/IKM swapped)
• ❌ MEDIUM: No signed prekey verification in X3DH
• ⚠️ Deviates from official Signal Protocol specification
• ⚠️ May have interoperability issues with other implementations

Status: Functional but requires specification compliance fixes

---

X3DH (Extended Triple Diffie-Hellman) Analysis

Protocol Overview

Location: src/rust/x3dh.rs (353 lines) Purpose: Asynchronous key agreement for initial session establishment Standard: Signal X3DH Specification

Implementation Flow

pub fn x3dh_initiate(
    alice_identity: &IdentityKeyPair,
    alice_ephemeral: &EphemeralKeyPair,
    bob_identity_public: &[u8],
    bob_signed_prekey_public: &[u8],
    bob_onetime_prekey_public: Option<&[u8]>,
) -> Result<X3DHResult, SignalError> {

    // DH1: alice_identity ⊗ bob_signed_prekey
    let dh1 = x25519_ecdh(&alice_identity.private_key, bob_signed_prekey_public)?;

    // DH2: alice_ephemeral ⊗ bob_identity
    let dh2 = x25519_ecdh(&alice_ephemeral.private_key, bob_identity_public)?;

    // DH3: alice_ephemeral ⊗ bob_signed_prekey
    let dh3 = x25519_ecdh(&alice_ephemeral.private_key, bob_signed_prekey_public)?;

    // DH4: alice_ephemeral ⊗ bob_onetime_prekey (if available)
    let dh4 = if let Some(bob_onetime) = bob_onetime_prekey_public {
        Some(x25519_ecdh(&alice_ephemeral.private_key, bob_onetime)?)
    } else {
        None
    };

    // ⚠️ ISSUE: HKDF parameter order
    let shared_secret = derive_x3dh_secret(&dh1, &dh2, &dh3, dh4.as_deref());

    Ok(X3DHResult {
        shared_secret,
        associated_data: Vec::new(),
    })
}

Security Properties

Property	Status	Evidence
Mutual authentication	✅ Correct	Identity keys involved in DH1, DH2
Forward secrecy	✅ Strong	Ephemeral keys in DH2, DH3, DH4
Deniability	✅ Present	No signatures on messages
Key confirmation	⚠️ Implicit	Via subsequent Double Ratchet
Replay protection	⚠️ Application layer	One-time prekey usage

DH Combination Analysis

Signal Specification Order:

DH1 = DH(IK_A, SPK_B)  # Alice identity × Bob signed prekey
DH2 = DH(EK_A, IK_B)   # Alice ephemeral × Bob identity
DH3 = DH(EK_A, SPK_B)  # Alice ephemeral × Bob signed prekey
DH4 = DH(EK_A, OPK_B)  # Alice ephemeral × Bob one-time prekey (optional)

Implementation Order: ✅ CORRECT (matches specification)

Critical Issue: No Signed Prekey Verification

Finding: Bob's signed prekey signature is never verified

Location: x3dh.rs:22-76 (x3dh_initiate function)

Impact:

• Man-in-the-middle attack possible
• Attacker could substitute malicious signed prekey
• Alice doesn't verify Bob's signature on SPK_B
• Breaks authentication property

Severity: 🟠 MEDIUM-HIGH

Signal Specification Requirement:

"Clients MUST verify the signed prekey signature before using it in the X3DH calculation"

Current Implementation: Signature is passed but never verified

Recommendation:

// Add before DH operations
ed25519_verify(
    bob_identity_public,
    bob_signed_prekey_public,
    bob_signed_prekey_signature
)?;

HKDF Derivation Issues

Location: x3dh.rs:183-196

fn derive_x3dh_secret(
    dh1: &[u8],
    dh2: &[u8],
    dh3: &[u8],
    dh4: Option<&[u8]>,
) -> Vec<u8> {
    let mut km = Vec::new();
    km.extend_from_slice(dh1);
    km.extend_from_slice(dh2);
    km.extend_from_slice(dh3);
    if let Some(dh4) = dh4 {
        km.extend_from_slice(dh4);
    }

    // ⚠️ ISSUE: Non-standard HKDF parameters
    let hk = Hkdf::<Sha256>::new(Some(b"X3DHSalt"), &km);
    let mut output = vec![0u8; 32];
    hk.expand(b"X3DHInfo", &mut output)
        .expect("HKDF expand failed");
    output
}

Signal Specification:

SK = KDF(DH1 || DH2 || DH3 || DH4)

Issue: Implementation uses custom salt ("X3DHSalt") and info ("X3DHInfo") not in specification

Impact:

• ❌ Not compatible with other Signal Protocol implementations
• ❌ Cannot interoperate with Signal app, libsignal, etc.
• ⚠️ Cryptographically secure but non-standard

Severity: 🟠 MEDIUM (Interoperability issue)

RFC Compliance Assessment

RFC Requirement	Status	Notes
4-DH computation	✅ Correct	All DH operations present
DH ordering	✅ Correct	Matches specification
HKDF usage	⚠️ Non-standard	Custom parameters
Signed prekey verification	❌ Missing	Critical gap
Associated data	✅ Present	Included in result
One-time prekey optional	✅ Correct	Properly handled

Overall X3DH Compliance: 🟡 66% (4/6 requirements met)

---

Double Ratchet Protocol Analysis

Protocol Overview

Location: src/rust/double_ratchet.rs (915 lines) Purpose: Forward secrecy and post-compromise security for ongoing messages Standard: Signal Double Ratchet Specification

Ratchet State Structure

pub struct RatchetState {
    pub dh_self: DHKeyPair,          // Own DH ratchet key
    pub dh_remote: Option<Vec<u8>>,  // Remote DH public key
    pub root_key: Vec<u8>,           // Root chain key (32 bytes)
    pub chain_key_send: Vec<u8>,     // Sending chain key (32 bytes)
    pub chain_key_recv: Vec<u8>,     // Receiving chain key (32 bytes)
    pub n_send: u32,                 // Sending message number
    pub n_recv: u32,                 // Receiving message number
    pub prev_chain_len: u32,         // Previous sending chain length
    pub skipped_keys: HashMap<(Vec<u8>, u32), Vec<u8>>, // Out-of-order messages
}

Assessment: ✅ All required state variables present

DH Ratchet Implementation

Location: double_ratchet.rs:216-257

fn dh_ratchet_step(
    state: &mut RatchetState,
    remote_public: &[u8],
) -> Result<(), SignalError> {

    // Advance receiving chain
    state.prev_chain_len = state.n_send;
    state.n_send = 0;
    state.n_recv = 0;

    // Perform DH with remote public key
    let dh_output = x25519_ecdh(&state.dh_self.private_key, remote_public)?;

    // ⚠️ ISSUE: HKDF parameter order reversed
    let hk = Hkdf::<Sha256>::new(Some(&state.root_key), &dh_output);

    // Derive new root key and chain key
    let mut new_root_key = vec![0u8; 32];
    let mut new_chain_key = vec![0u8; 32];

    hk.expand(b"RootKey", &mut new_root_key)
        .map_err(|_| SignalError::KdfFailed)?;
    hk.expand(b"ChainKey", &mut new_chain_key)
        .map_err(|_| SignalError::KdfFailed)?;

    state.root_key = new_root_key;
    state.chain_key_send = new_chain_key;
    state.dh_remote = Some(remote_public.to_vec());

    // Generate new DH key pair for next ratchet
    state.dh_self = generate_dh_keypair()?;

    Ok(())
}

Security Properties:

• ✅ Correct DH ratchet advancement
• ✅ Root key properly updated
• ✅ New ephemeral key generated
• ⚠️ HKDF parameter order issue (see below)

Symmetric Ratchet Implementation

Location: double_ratchet.rs:259-296

fn symmetric_ratchet_step(
    chain_key: &[u8],
) -> (Vec<u8>, Vec<u8>) {

    // ⚠️ ISSUE: HKDF parameter order
    let hk = Hkdf::<Sha256>::new(Some(b"ChainKeySalt"), chain_key);

    // Derive next chain key
    let mut next_chain_key = vec![0u8; 32];
    hk.expand(b"ChainKey", &mut next_chain_key)
        .expect("HKDF expand failed");

    // Derive message key
    let mut message_key = vec![0u8; 32];
    hk.expand(b"MessageKey", &mut message_key)
        .expect("HKDF expand failed");

    (next_chain_key, message_key)
}

Assessment:

• ✅ Correct ratchet advancement logic
• ✅ Separate message keys per message
• ⚠️ Non-standard HKDF parameters

Critical Issue: AAD Not Used in Encryption

Finding: AES-GCM encryption does not use Additional Authenticated Data (AAD)

Location: double_ratchet.rs:525, 657

Current Implementation:

fn encrypt_message(
    message_key: &[u8],
    plaintext: &[u8],
    nonce: &[u8],
) -> Result<Vec<u8>, SignalError> {

    let cipher = Aes256Gcm::new(Key::<Aes256Gcm>::from_slice(message_key));
    let nonce_array = GenericArray::from_slice(nonce);

    // ⚠️ CRITICAL: No AAD parameter
    let ciphertext = cipher
        .encrypt(nonce_array, plaintext)  // Should include AAD!
        .map_err(|_| SignalError::EncryptionFailed)?;

    Ok(ciphertext)
}

Signal Specification:

// Should include message header as AAD
let aad = encode_message_header(dh_public, prev_chain_len, message_number);
cipher.encrypt(nonce, Payload { msg: plaintext, aad: &aad })

Impact:

• ❌ Message metadata not authenticated
• ❌ Message reordering attacks possible
• ❌ No cryptographic binding between header and ciphertext
• ❌ Violates Signal Protocol specification

Severity: 🔴 CRITICAL

Recommendation:

fn encrypt_message(
    message_key: &[u8],
    plaintext: &[u8],
    nonce: &[u8],
    message_number: u32,        // Add parameter
    prev_chain_length: u32,     // Add parameter
) -> Result<Vec<u8>, SignalError> {

    let cipher = Aes256Gcm::new(Key::<Aes256Gcm>::from_slice(message_key));
    let nonce_array = GenericArray::from_slice(nonce);

    // Create AAD from message metadata
    let mut aad = Vec::new();
    aad.extend_from_slice(&message_number.to_le_bytes());
    aad.extend_from_slice(&prev_chain_length.to_le_bytes());

    // Use AAD in encryption
    let payload = Payload {
        msg: plaintext,
        aad: &aad,
    };

    let ciphertext = cipher
        .encrypt(nonce_array, payload)
        .map_err(|_| SignalError::EncryptionFailed)?;

    Ok(ciphertext)
}

Critical Issue: HKDF Parameter Order

Finding: Salt and IKM parameters are consistently reversed throughout

Location: double_ratchet.rs:355-358 and multiple other locations

RFC 5869 Specification:

HKDF-Extract(salt, IKM) -> PRK

Current Implementation:

// ⚠️ ISSUE: Parameters reversed
let hk = Hkdf::<Sha256>::new(
    Some(salt),   // First parameter should be optional salt
    ikm           // Second parameter should be IKM
);
// But code uses: new(Some(ikm), salt) in many places

Signal Specification:

KDF(KM) = HKDF(salt=F || KM, IKM=constant)

Impact:

• ❌ Deviates from RFC 5869 intended usage
• ❌ Not compatible with other Signal implementations
• ⚠️ Still cryptographically secure (HKDF extract is commutative)
• ❌ Interoperability broken

Severity: 🟡 MEDIUM (Correctness issue, not security vulnerability)

Out-of-Order Message Handling

Location: double_ratchet.rs:470-523

fn try_skipped_message_keys(
    state: &mut RatchetState,
    header: &MessageHeader,
    ciphertext: &[u8],
) -> Result<Option<Vec<u8>>, SignalError> {

    let key_id = (header.dh_public_key.clone(), header.message_number);

    if let Some(message_key) = state.skipped_keys.get(&key_id) {
        // Decrypt with skipped key
        let plaintext = decrypt_message(message_key, ciphertext, &header.nonce)?;

        // ✅ SECURITY: Remove used key immediately
        state.skipped_keys.remove(&key_id);

        return Ok(Some(plaintext));
    }

    Ok(None)
}

Security Analysis:

• ✅ Skipped keys stored correctly
• ✅ Keys deleted after use (prevents replay)
• ✅ Bounded storage (MAX_SKIP = 1000)
• ✅ Proper key identification

Max Skip Limit: 1000 messages

• Prevents DoS via memory exhaustion
• Reasonable for most use cases
• ✅ Industry standard

Security Properties Verification

Property	Status	Evidence
Forward Secrecy	✅ Strong	DH ratchet + ephemeral keys
Post-Compromise Security	✅ Strong	DH ratchet heals compromise
Message Confidentiality	✅ Strong	AES-256-GCM
Message Authenticity	⚠️ Partial	GCM tag but no AAD
Out-of-order tolerance	✅ Correct	Skipped key storage
Message loss tolerance	✅ Correct	Independent message keys
Break-in recovery	✅ Present	Next DH ratchet step

---

Attack Scenario Analysis

Scenario 1: Message Reordering Attack

Enabled by: AAD not used in AES-GCM

Attack:

1. Attacker intercepts messages M1, M2, M3
2. Delivers in wrong order: M2, M1, M3
3. No cryptographic binding to prevent this
4. Application may process out-of-order

Current Protection: ❌ None (message number not authenticated)

Fix: Include message number in AAD

Severity: 🔴 HIGH

Scenario 2: Man-in-the-Middle on X3DH

Enabled by: No signed prekey verification

Attack:

1. Attacker intercepts Bob's signed prekey
2. Substitutes attacker's own key
3. Alice completes X3DH with attacker's key
4. Attacker can decrypt session

Current Protection: ❌ None (signature never checked)

Fix: Verify ed25519 signature on signed prekey

Severity: 🟠 HIGH

Scenario 3: Replay Attack

Enabled by: Protocol design (not implementation flaw)

Attack:

1. Attacker captures encrypted message
2. Replays to recipient multiple times
3. Each replay attempts decryption

Current Protection: ✅ Partial

• Used keys removed from skipped_keys
• Prevents replay of out-of-order messages
• ⚠️ In-order messages could be replayed if application doesn't track

Recommendation: Application-level sequence number tracking

Severity: 🟡 MEDIUM (Application responsibility)

---

Specification Compliance Summary

X3DH Compliance

Requirement	Status	Location
Identity key DH	✅ Correct	x3dh.rs:51-55
Ephemeral key DH	✅ Correct	x3dh.rs:58-69
Signed prekey DH	✅ Correct	x3dh.rs:51-55, 65-69
One-time prekey DH	✅ Correct	x3dh.rs:72-76
Signed prekey verify	❌ Missing	Critical gap
HKDF derivation	⚠️ Non-standard	x3dh.rs:183-196
Associated data	✅ Present	x3dh.rs:113-118
Deniability	✅ Correct	No signatures on messages

Overall X3DH Compliance: 🟡 62.5% (5/8 requirements fully met)

Double Ratchet Compliance

Requirement	Status	Location
DH ratchet step	✅ Correct	double_ratchet.rs:216-257
Symmetric ratchet	✅ Correct	double_ratchet.rs:259-296
Root key update	✅ Correct	double_ratchet.rs:234-243
Chain key update	✅ Correct	double_ratchet.rs:268-272
Message key derivation	✅ Correct	double_ratchet.rs:274-278
AAD in encryption	❌ Missing	Critical gap
Out-of-order handling	✅ Correct	double_ratchet.rs:470-523
Skipped key storage	✅ Correct	double_ratchet.rs:483-490
HKDF parameters	⚠️ Non-standard	Throughout

Overall Double Ratchet Compliance: 🟡 66% (6/9 requirements fully met)

---

Recommendations

Critical (P0) - Fix Before Production

1. Implement AAD in AES-GCM encryption

   • Add message number and chain length as AAD
   • Prevents message reordering attacks
   • Effort: 4-6 hours

2. Add signed prekey verification in X3DH

   • Verify Ed25519 signature before use
   • Prevents MITM attacks
   • Effort: 1-2 hours

High (P1) - Within 2 Weeks

3. Fix HKDF parameter order

   • Swap salt and IKM to match RFC 5869
   • Update all HKDF calls throughout codebase
   • WARNING: Breaks compatibility with existing sessions
   • Effort: 6-8 hours

4. Update to standard X3DH derivation

   • Remove custom salt/info parameters
   • Use Signal specification exactly
   • Enables interoperability
   • Effort: 2-3 hours

Medium (P2) - Within 1 Month

5. Add key confirmation step

   • Explicit verification both parties have same key
   • Prevents subtle MITM scenarios
   • Effort: 3-4 hours

6. Implement application-level replay protection

   • Track message sequence numbers
   • Reject duplicate or old messages
   • Effort: 4-6 hours

---

Comparison with MLS Protocol

Aspect	Signal (Double Ratchet)	MLS (TreeKEM)
Group size	Optimal for 1:1	Optimal for groups
Forward secrecy	Per-message	Per-epoch
PCS recovery	Immediate (next DH)	Next commit
Overhead	Low	Higher (tree updates)
Out-of-order	Excellent	Limited
Message loss	Tolerant	Requires recovery
Implementation	⚠️ Specification deviations	✅ RFC 9420 compliant

Verdict: Signal is better for 1:1 messaging, MLS for group communication

---

Conclusion

Protocol Security Assessment: 🟡 MEDIUM

Strengths:

• ✅ Core cryptographic flows are correct
• ✅ Forward secrecy properly implemented
• ✅ Post-compromise security functional
• ✅ Out-of-order message handling works
• ✅ Skipped key management secure

Critical Gaps:

• ❌ AAD not used in AES-GCM encryption
• ❌ Signed prekey signature never verified
• ⚠️ HKDF parameter order reversed
• ⚠️ Non-standard derivation parameters

Risk Level: 🟡 MEDIUM

Verdict: Functional and cryptographically secure in isolation, but not compatible with other Signal Protocol implementations and missing specification-required security checks.

Recommendation: Fix P0 issues (AAD, signed prekey verification) before production deployment. P1 fixes improve interoperability but may break existing sessions.

---

Document Version: 1.0 Last Updated: January 2025 Next Review: After specification compliance fixes