WDP Part 9a: Catalog Format Specification

Abstract#

This document specifies the catalog file format for the Waddling Diagnostic Protocol (WDP). Catalogs constitute the mechanism by which compact identifiers are mapped to complete diagnostic metadata, enabling efficient distribution and client-side expansion. This specification defines two catalog types (single-namespace and aggregated), three format variants (full, compact, and minimal), and validation requirements.

Part 9a focuses on the catalog file structure. Part 9b 9b-WIRE-PROTOCOL.md specifies the runtime wire protocol. Part 9c 9c-IMPLEMENTATION.md provides implementation guidance.

Specification Navigation: Refer to STRUCTURE.md for an overview of all WDP specification documents.

Status of This Memo#

This document specifies a standards track protocol for the WDP community and requests discussion and suggestions for improvements. Distribution of this memo is unlimited.

Requirements Language#

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

1. Introduction#

1.1 Purpose#

WDP catalogs are JSON files that map compact identifiers to diagnostic metadata. Catalogs enable:

• Efficient Transmission: Compact identifiers are transmitted instead of full messages
• Offline Capability: Clients cache catalogs locally for offline operation
• Multi-Language Support: Identical compact identifiers across language-specific catalogs
• Fast Lookups: O(1) hash-based retrieval for both single-namespace and aggregated catalogs
• Namespace Support: Collision-free aggregation of multiple services and libraries

1.2 Architecture#

┌─────────────────────────────────────────────────┐
│  Build Time                                     │
│  ┌────────────────┐                             │
│  │ Error Registry │                             │
│  │ (Source Code)  │                             │
│  └────────┬───────┘                             │
│           │                                     │
│           ▼                                     │
│  ┌────────────────┐                             │
│  │ Generate       │                             │
│  │ Catalog JSON   │                             │
│  └────────┬───────┘                             │
87: └───────────┼─────────────────────────────────────┘
            │
            ▼
┌─────────────────────────────────────────────────┐
│  Distribution                                   │
│  ┌────────────────┐                             │
│  │ catalog.json   │                             │
│  │ (50KB)         │                             │
│  └────────┬───────┘                             │
│           │                                     │
│           ▼                                     │
│  ┌────────────────────────────────────┐         │
│  │ CDN / Static File Server           │         │
│  └────────────────────────────────────┘         │
└─────────────────────────────────────────────────┘

1.3 Design Goals#

• Minimal Overhead: Catalog file size SHOULD remain small
• Fast Parsing: Simple JSON structure for efficient parsing
• Format Flexibility: Multiple formats for different use cases
• Versioning: Support for catalog evolution
• Language Agnosticism: Pure JSON without language-specific features

1.4 Conformance Levels#

An implementation is considered conformant at Level 2 (Extended) if it correctly implements all requirements marked with MUST, REQUIRED, or SHALL in this specification, in addition to achieving Level 0 (Error Codes) and Level 1 (Compact IDs) conformance.

2. Catalog Purpose and Use Cases#

2.1 Primary Use Cases#

2.1.1 Constrained Devices (IoT)#

Problem: Constrained bandwidth, battery, and storage

Solution:

Sensor → {"xY9Kp":{"f":{"temp":"45.2"}}} → Gateway

Gateway expands using catalog:
"Temperature 45.2°C exceeds threshold"

Benefits:

• Efficient transmission with minimal overhead
• Reduced battery consumption
• Local catalog caching on gateway

2.1.2 Mobile Applications#

Problem: Slow or metered mobile networks, offline scenarios

Solution:

1. Application downloads catalog once (cached locally)
2. API returns compact diagnostics
3. Application expands diagnostics locally (offline-capable)

Benefits:

• Fast API responses
• Local message resolution without server round-trips
• Multiple language support (download appropriate catalog)

2.1.3 Microservices Architecture#

Problem: High-volume logging across distributed services

Solution:

Service A logs: [xY9Kp] Auth failure
Service B logs: [xY9Kp] Auth failure
Service C logs: [xY9Kp] Auth failure

Centralized logging searches by compact ID
Aggregates all instances of identical diagnostic

Benefits:

• Compact log entries
• Straightforward aggregation by compact ID
• Centralized diagnostic analysis

2.1.4 Multi-Language Applications#

Problem: Supporting multiple languages efficiently

Solution:

Backend → {"xY9Kp":{"f":{"timestamp":"2024-01-15"}}}

English client: "Token expired at 2024-01-15"
Spanish client: "Token expiró el 2024-01-15"
Japanese client: "トークンの有効期限が切れました 2024-01-15"

Benefits:

• Identical compact ID for all languages
• Clients download appropriate language catalog
• Backend code requires no internationalization logic

2.2 Non-Goals#

WDP catalogs are NOT intended for:

• Authentication/Authorization: Compact identifiers are not cryptographic tokens
• Real-time Catalog Updates: Catalogs are versioned artifacts, not live configuration
• Custom Business Logic: Catalogs store metadata, not executable code
• Large Binary Data: Catalogs SHOULD remain small (less than 500KB RECOMMENDED)

3. Catalog Types#

WDP defines two catalog types based on scope. Each type MAY use any of the three format variants (Full, Compact, Minimal).

3.1 Single-Namespace Catalog#

Purpose: Default catalog type for most clients

Characteristics:

• Keys in diags MUST be CompactID format (5 alphanumeric characters)
• Represents diagnostics from a single service, library, or application
• MAY include optional namespace and namespace_hash fields for documentation
• MUST NOT include namespaces index field
• MUST NOT use CombinedID format in keys

Applicability:

• Mobile applications (single backend)
• Web applications (single API)
• Individual microservices
• IoT devices (typical case)
• Single library or SDK
• Standalone services

Basic Structure:

{
  "version": "1.0.0",
  "diags": {
    "81E9g": { ... },
    "xY9Kp": { ... }
  }
}

With Optional Namespace Metadata:

{
  "version": "1.0.0",
  "namespace": "auth_service",
  "namespace_hash": "h4tYw2",
  "diags": {
    "81E9g": { ... },
    "xY9Kp": { ... }
  }
}

Key Format: ^[A-Za-z0-9]{5}$

3.2 Aggregated Catalog#

Purpose: Catalog combining multiple namespaces

Characteristics:

• Keys in diags MUST be CombinedID format (<NamespaceHash>-<CompactID>)
• Represents diagnostics from multiple services, libraries, or sources
• MAY include namespaces index mapping names to hashes
• MUST NOT include top-level namespace or namespace_hash fields
• MUST NOT use simple CompactID format in keys

Applicability:

• Monitoring dashboards (multiple services)
• Edge gateways (multiple devices)
• Log aggregators (multiple sources)
• SDK bundles (library with dependencies)
• Operations and observability tools
• Multi-tenant platforms
• API gateways and BFFs

Basic Structure (Privacy-Preserving):

{
  "version": "1.0.0",
  "diags": {
    "h4tYw2-81E9g": { ... },
    "k9Px3a-xY9Kp": { ... }
  }
}

With Namespaces Index (Human-Readable):

{
  "version": "1.0.0",
  "namespaces": {
    "auth_service": "h4tYw2",
    "payment_service": "k9Px3a"
  },
  "diags": {
    "h4tYw2-81E9g": { ... },
    "k9Px3a-xY9Kp": { ... }
  }
}

Key Format: ^[A-Za-z0-9]{5}-[A-Za-z0-9]{5}$

Benefits:

• Collision-Free: Different namespaces MAY have identical error codes
• Privacy-Preserving: namespaces index MAY be omitted to hide service names
• Stable: Namespace hash remains constant regardless of service renaming
• O(1) Lookup: Direct key access regardless of aggregation

3.3 Catalog Type Selection#

Decision Tree:

Does client receive diagnostics from multiple services or libraries?
  ├─ NO  → Use Single-Namespace Catalog
  └─ YES → Use Aggregated Catalog
           ├─ Require human-readable names? → Include namespaces index
           └─ Privacy-sensitive? → Omit namespaces index

3.4 Design Rationale: Flat Structure#

Question: Why use CombinedID keys instead of nested namespace objects?

The flat structure with CombinedID keys ("h4tYw2-81E9g": {...}) was selected over nested structures ({"h4tYw2": {"diags": {"81E9g": {...}}}}) for the following reasons:

3.4.1 Performance#

• O(1) Lookup: Direct key access catalog.diags[combinedId] without parsing or navigation
• No Nested Iteration: Single hash map lookup regardless of namespace count
• Minimal Overhead: Only 6 additional characters per key (namespace hash + hyphen)

3.4.2 Implementation Simplicity#

• Uniform Lookup Pattern: Both single-namespace and aggregated catalogs use identical access: catalog.diags[id]
• Consistent Client Code: Identical lookup logic regardless of catalog type
• Straightforward Validation: Single key format verification across all entries

3.4.3 Size Efficiency#

• Compact Structure: No nesting overhead, repetitive field names, or structural wrappers
• Improved Compression: Flat JSON structures compress more efficiently
• Minimal Overhead: CombinedID format adds only namespace hash prefix to keys

3.4.4 Aggregation Benefits#

• Trivial Merging: Concatenate diags objects directly: {...catalog1.diags, ...catalog2.diags}
• No Restructuring: Source catalogs MAY be flat; aggregation preserves structure
• Build-Time Efficiency: Simple key transformation (compactId → nsHash-compactId)

4. Catalog Format Variants#

Each catalog type (Single-Namespace or Aggregated) MAY use any of three format variants.

4.1 Full Format#

Purpose: Development, debugging, documentation

Characteristics:

• All metadata fields included
• Human-readable field names
• Complete documentation
• Suitable for development and documentation

Applicability:

• Development environments
• API documentation
• Error reference guides
• Internal tools

Structure:

{
  "version": "1.0.0",
  "generated": "2024-01-15T10:30:00Z",
  "diags": {
    "<compact_id>": {
      "code": "<full_wdp_code>",
      "severity": "<severity_char>",
      "message": "<diagnostic_message>",
      "description": "<detailed_explanation>",
      "hints": ["<suggestion_1>", "<suggestion_2>"],
      "tags": ["<tag_1>", "<tag_2>"],
      "fields": ["<field_1>", "<field_2>"]
    }
  }
}

Example:

{
  "version": "1.0.0",
  "generated": "2024-01-15T10:30:00Z",
  "diags": {
    "jGKFp": {
      "code": "E.AUTH.TOKEN.001",
      "severity": "E",
      "message": "Token missing from Authorization header",
      "description": "The JWT token was not provided in the request. Authentication requires a valid Bearer token in the Authorization header.",
      "hints": [
        "Include header: Authorization: Bearer <token>",
        "Obtain token from /auth/login endpoint"
      ],
      "tags": ["auth", "security", "http"],
      "fields": []
    },
    "xY9Kp": {
      "code": "E.AUTH.TOKEN.EXPIRED",
      "severity": "E",
      "message": "Token expired at {{timestamp}}",
      "description": "The JWT token has exceeded its time-to-live (TTL). Please request a new token using your refresh token.",
      "hints": [
        "Use /auth/refresh endpoint with refresh token",
        "Check token expiration time (exp claim)"
      ],
      "tags": ["auth", "security", "jwt"],
      "fields": ["timestamp"]
    }
  }
}

4.2 Compact Format#

Purpose: Production, network transmission

Characteristics:

• Abbreviated field names (1-2 characters)
• Essential fields only
• Optimized for network transmission

Applicability:

• Production web applications
• Mobile applications
• API responses
• General-purpose catalogs

Structure:

{
  "v": "<version>",
  "g": "<generated_timestamp>",
  "wd": {
    "<compact_id>": {
      "c": "<full_wdp_code>",
      "s": "<severity_char>",
      "m": "<message>",
      "d": "<description>",
      "h": ["<hint_1>", "<hint_2>"],
      "t": ["<tag_1>", "<tag_2>"],
      "f": ["<field_1>", "<field_2>"]
    }
  }
}

Field Mapping:

Example:

{
  "v": "1.0.0",
  "g": "2024-01-15T10:30:00Z",
  "wd": {
    "jGKFp": {
      "c": "E.AUTH.TOKEN.001",
      "s": "E",
      "m": "Token missing from Authorization header",
      "d": "The JWT token was not provided in the request.",
      "h": ["Include header: Authorization: Bearer <token>"],
      "t": ["auth", "security"],
      "f": []
    },
    "xY9Kp": {
      "c": "E.AUTH.TOKEN.EXPIRED",
      "s": "E",
      "m": "Token expired at {{timestamp}}",
      "d": "The JWT token has exceeded its TTL.",
      "h": ["Use /auth/refresh endpoint"],
      "t": ["auth", "jwt"],
      "f": ["timestamp"]
    }
  }
}

4.3 Minimal Format#

Purpose: Extremely constrained environments (IoT, embedded systems)

Characteristics:

• Array-based structure without field names
• Code and message only
• Minimalist structure for constrained environments

Applicability:

• IoT devices with storage constraints
• Embedded systems
• Specialized protocols

Structure:

{
  "<compact_id>": ["<code>", "<message>"],
  "<compact_id>": ["<code>", "<message>"]
}

Example:

{
  "jGKFp": ["E.AUTH.TOKEN.001", "Token missing from Authorization header"],
  "xY9Kp": ["E.AUTH.TOKEN.EXPIRED", "Token expired at {{timestamp}}"],
  "mN3Yr": ["W.DATABASE.CONNECTION.027", "Database connection pool near capacity ({{current}}/{{max}})"]
}

Array Position:

4.4 Format Selection Guide#

Recommendation: Begin with Compact format for most production applications. Use Full format for documentation and development. Reserve Minimal format for truly constrained environments.

5. Field Specifications#

5.1 Hash Algorithm#

NORMATIVE: The hash algorithm for compact identifiers is fixed to xxHash3 (64-bit variant) with seed 0x000031762D706477 as specified in 5-COMPACT-IDS.md. This value is immutable for WDP v1 and MUST NOT be included in catalog files.

Rationale: Algorithm specification in catalogs would create versioning complexity and interoperability issues. A fixed algorithm ensures universal compatibility.

5.2 Namespace Fields#

These fields enable single-namespace documentation and multi-namespace aggregation.

5.2.1 namespace (OPTIONAL)#

Type: String
Format: Lowercase namespace identifier
Pattern: ^[a-z][a-z0-9_]{0,31}$
Example: "auth_service", "payment_lib", "my_app"

Purpose: Human-readable namespace name for single-namespace catalogs (documentation only)

Requirements:

• MAY be included in single-namespace catalogs for documentation
• MUST NOT be included in aggregated catalogs (use namespaces index instead)
• MUST NOT be present if any diags key contains a hyphen (CombinedID format)
• If present, diags keys MUST be CompactID format (5 characters, no hyphen)

Validation:

• MUST begin with lowercase letter
• MUST contain only lowercase letters, digits, and underscores
• MUST be 1-32 characters in length

Compact Key: ns

Example:

{
  "version": "1.0.0",
  "namespace": "auth_service",
  "diags": {
    "81E9g": { ... }
  }
}

5.2.2 namespace_hash (OPTIONAL)#

Type: String
Format: 5-character Base62 namespace hash
Pattern: ^[A-Za-z0-9]{5}$
Example: "h4tYw2", "k9Px3a"

Purpose: Stable hash of the namespace for verification and cross-reference

Requirements:

• MAY be included in single-namespace catalogs alongside namespace
• MUST NOT be included in aggregated catalogs
• MUST NOT be present if any diags key contains a hyphen
• If present, SHOULD match the namespace hash used in combined identifiers elsewhere
• Hash MUST be generated using xxHash3 with seed 0x762D736E2D706477 as specified in 7-NAMESPACES.md

Applicability:

• Catalog integrity verification
• Cross-reference with aggregated catalogs
• Tooling and automation

Compact Key: nsh

Example:

{
  "version": "1.0.0",
  "namespace": "auth_service",
  "namespace_hash": "h4tYw2",
  "diags": {
    "81E9g": { ... }
  }
}

5.2.3 namespaces (OPTIONAL)#

Type: Object (hash map)
Keys: Namespace strings
Values: 5-character Base62 namespace hashes
Example: {"auth_service": "h4tYw2", "payment_lib": "k9Px3a"}

Purpose: Human-readable index mapping namespace names to hashes in aggregated catalogs

Requirements:

• MUST be included in aggregated catalogs if human-readable names are desired
• MAY be omitted for privacy-preserving catalogs (hashes only)
• MUST NOT be included in single-namespace catalogs
• If present, at least one diags key MUST be in CombinedID format (contain hyphen)
• All namespace hashes appearing in diags keys SHOULD have corresponding entries

Validation:

• Keys MUST be valid namespace strings (match namespace pattern)
• Values MUST be valid namespace hashes (5-character Base62)
• For each CombinedID in diags, the namespace hash portion SHOULD appear in namespaces values

Compact Key: nss

Benefits:

• Enables reverse lookup (hash to name)
• Provides documentation and context
• Supports tooling and monitoring
• Human-readable error attribution

Example (Aggregated with Index):

{
  "version": "1.0.0",
  "namespaces": {
    "auth_service": "h4tYw2",
    "payment_lib": "k9Px3a",
    "user_service": "m7Nq4r"
  },
  "diags": {
    "h4tYw2-81E9g": { ... },
    "k9Px3a-xY9Kp": { ... },
    "m7Nq4r-aB3Cd": { ... }
  }
}

Example (Privacy-Preserving, No Index):

{
  "version": "1.0.0",
  "diags": {
    "h4tYw2-81E9g": { ... },
    "k9Px3a-xY9Kp": { ... }
  }
}

Reverse Lookup Algorithm:

function getNamespace(combinedId: string, catalog: Catalog): string {
  const nsHash = combinedId.split('-')[0];
  
  if (!catalog.namespaces) {
    return nsHash;  // Privacy mode, return hash
  }
  
  for (const [name, hash] of Object.entries(catalog.namespaces)) {
    if (hash === nsHash) return name;
  }
  
  return 'unknown';
}

5.3 Core Catalog Fields#

5.3.1 wdp_version (OPTIONAL)#

Type: String
Format: WDP specification version (MAJOR.MINOR)
Example: "1.0", "2.0"

Purpose: Declares which WDP specification version was used to generate this catalog

Requirements:

• MUST include for catalogs supporting multiple WDP versions (transition catalogs)
• MUST include for empty catalogs (no compact identifiers to infer from)
• SHOULD include for WDP v2.0+ catalogs (for clarity)
• MAY omit for pure WDP v1.0 catalogs with diagnostic entries present

Version Inference:

When wdp_version field is present:

• Use the declared version
• Validate that compact identifiers match declared version format

When wdp_version field is absent:

• Infer from compact identifier formats:
  • All identifiers match [0-9A-Za-z]{5} → WDP v1.0
  • All identifiers match 2\.[0-9A-Za-z]+ → WDP v2.0
  • If empty or mixed formats → field is REQUIRED

Validation:

• MUST match format: ^\d+\.\d+$
• MUST be a valid WDP version ("1.0", "2.0", etc.)

Compact Key: wv

5.3.2 version (REQUIRED)#

Type: String
Format: Semantic version (MAJOR.MINOR.PATCH)
Example: "1.0.0", "2.3.1"

Purpose: Catalog version for change tracking and cache invalidation

Validation:

• MUST follow semantic versioning (semver.org)
• MUST be unique for each catalog release
• SHOULD increment on any change

Compact Key: v

5.3.3 generated (OPTIONAL)#

Type: String
Format: ISO 8601 timestamp
Example: "2024-01-15T10:30:00Z"

Purpose: Timestamp indicating when catalog was generated

Applicability:

• Debugging stale catalog issues
• Audit trails
• Cache validation

Compact Key: g

5.3.4 diags (REQUIRED)#

Type: Object (hash map)
Keys: CompactID or CombinedID (depends on catalog type)
Values: Diagnostic metadata objects

Purpose: The main catalog data structure containing diagnostics of all severity types

Key Format Rules:

For Single-Namespace Catalogs:

• Keys MUST be CompactID format: ^[A-Za-z0-9]{5}$
• Example: "81E9g", "xY9Kp"
• All keys MUST be 5 characters, no hyphens
• MUST NOT mix with CombinedID format

For Aggregated Catalogs:

• Keys MUST be CombinedID format: ^[A-Za-z0-9]{5}-[A-Za-z0-9]{5}$
• Example: "h4tYw2-81E9g", "k9Px3a-xY9Kp"
• Format: <NamespaceHash>-<CompactID>
• All keys MUST be 11 characters with hyphen separator
• MUST NOT mix with simple CompactID format

Validation:

• MUST be an object
• Keys MUST match either CompactID OR CombinedID pattern (not mixed)
• Values MUST be valid diagnostic metadata objects
• If ANY key contains -, ALL keys MUST be CombinedID format
• If ALL keys are 5 characters, ALL MUST be CompactID format

Lookup Performance: Both formats provide O(1) direct key access. No iteration or parsing required for lookups.

Compact Key: wd

Example (Single-Namespace):

{
  "diags": {
    "81E9g": {"code": "E.Auth.Token.001", ...},
    "xY9Kp": {"code": "E.Auth.Token.018", ...}
  }
}

Example (Aggregated):

{
  "namespaces": {"auth_service": "h4tYw2", "payment_lib": "k9Px3a"},
  "diags": {
    "h4tYw2-81E9g": {"code": "E.Auth.Token.001", ...},
    "k9Px3a-xY9Kp": {"code": "E.Payment.Card.003", ...}
  }
}

5.4 Diagnostic Metadata Fields#

Each diagnostic entry in the catalog MUST include the following fields:

5.4.1 code (REQUIRED)#

Type: String
Format: Valid WDP structured code
Example: "E.AUTH.TOKEN.001"

Purpose: Full WDP structured code for reference

Validation:

• MUST be valid WDP structured code format: SEVERITY.COMPONENT.PRIMARY.SEQUENCE
• MUST conform to Part 1 (Severity), Part 2 (Component), Part 3 (Primary), and Part 4 (Sequence) specifications

Compact Key: c

5.4.2 severity (REQUIRED)#

Type: String
Format: Single character
Example: "E", "W", "C", "I", "H", "B", "S", "K", "T"

Purpose: Diagnostic severity level

Valid Values:

Validation:

• MUST be exactly one character
• MUST match first character of code field
• MUST be one of: E, B, C, W, H, S, K, I, T

Compact Key: s

5.4.3 message (REQUIRED)#

Type: String
Example: "Token expired at {{timestamp}}"

Purpose: Human-readable diagnostic message template

Validation:

• SHOULD be concise (fewer than 200 characters)
• MAY include field placeholders in {{field_name}} format
• MAY include PII placeholders in {{pii/field_name}} format

Placeholder Types:

Messages support two categories of placeholders:

1. Standard Placeholders: {{field_name}}
   • Referenced in the fields array
   • Values provided in wire protocol f object
   • Example: {{timestamp}}, {{error_code}}
2. PII Placeholders: {{pii/field_name}} or {{pii/field_name:suffix}}
   • Contain personally identifiable information
   • Values provided in wire protocol pii.data object
   • Support format suffixes for security handling
   • Example: {{pii/user_email}}, {{pii/ip_address:raw}}

For complete PII wire protocol requirements, see Part 9b Section 4.6.

Compact Key: m

5.4.4 description (OPTIONAL)#

Type: String
Example: "The JWT token has exceeded its time-to-live (TTL)."

Purpose: Detailed explanation of the diagnostic

Validation: SHOULD provide context beyond message, 1-3 sentences.

Compact Key: d

5.4.5 hints (OPTIONAL)#

Type: Array of strings
Example: ["Use /auth/refresh endpoint", "Check token expiration"]

Purpose: Actionable suggestions for resolving the issue

Validation: Each hint SHOULD be actionable and concise. RECOMMENDED 1-5 hints.

Compact Key: h

5.4.6 tags (OPTIONAL)#

Type: Array of strings
Example: ["auth", "security", "jwt"]

Purpose: Categorization and filtering

Validation: SHOULD be lowercase, single words or hyphenated. RECOMMENDED 1-5 tags.

Compact Key: t

5.4.7 fields (OPTIONAL)#

Type: Array of strings
Example: ["timestamp", "user_id"]

Purpose: List of field names used in message placeholders

Validation:

• Each field name MUST match pattern: [a-zA-Z_][a-zA-Z0-9_]*
• Field names MUST correspond to placeholders in message
• Order is not significant

Compact Key: f

5.5 Field Requirements by Format#

6. Validation#

6.1 Catalog Type Validation#

Catalogs MUST be validated to ensure conformance to either single-namespace or aggregated patterns.

6.1.1 Rule 1: Key Format Consistency#

Implementations MUST NOT mix CompactID and CombinedID formats:

def validate_key_consistency(diags):
    has_hyphen = any('-' in key for key in diags.keys())
    no_hyphen = all('-' not in key for key in diags.keys())
    
    if not (has_hyphen or no_hyphen):
        raise ValidationError("Cannot mix CompactID and CombinedID formats")
    
    if has_hyphen:
        # All keys must be CombinedID (11 chars with hyphen)
        for key in diags.keys():
            if not re.match(r'^[A-Za-z0-9]{5}-[A-Za-z0-9]{5}$', key):
                raise ValidationError(f"Invalid CombinedID format: {key}")
    else:
        # All keys must be CompactID (5 chars)
        for key in diags.keys():
            if not re.match(r'^[A-Za-z0-9]{5}$', key):
                raise ValidationError(f"Invalid CompactID format: {key}")

Valid Examples:

// All CompactID
{
  "diags": {
    "81E9g": {...},
    "xY9Kp": {...}
  }
}

// All CombinedID
{
  "diags": {
    "h4tYw2-81E9g": {...},
    "k9Px3a-xY9Kp": {...}
  }
}

Invalid Example:

// Mixed formats - INVALID
{
  "diags": {
    "81E9g": {...},           // CompactID
    "h4tYw2-xY9Kp": {...}     // CombinedID - ERROR
  }
}

6.1.2 Rule 2: Namespace Field Constraints#

Single-Namespace Catalogs (CompactID keys):

• MAY include namespace field
• MAY include namespace_hash field
• MUST NOT include namespaces field

Aggregated Catalogs (CombinedID keys):

• MUST NOT include namespace field
• MUST NOT include namespace_hash field
• MAY include namespaces field

def validate_namespace_fields(catalog):
    has_combined_ids = any('-' in key for key in catalog['diags'].keys())
    
    if has_combined_ids:
        # Aggregated catalog
        if 'namespace' in catalog:
            raise ValidationError(
                "Aggregated catalog MUST NOT include 'namespace' field"
            )
        if 'namespace_hash' in catalog:
            raise ValidationError(
                "Aggregated catalog MUST NOT include 'namespace_hash' field"
            )
    else:
        # Single-namespace catalog
        if 'namespaces' in catalog:
            raise ValidationError(
                "Single-namespace catalog MUST NOT include 'namespaces' field"
            )

6.1.3 Rule 3: Referential Integrity#

If namespaces index exists, implementations SHOULD validate namespace hashes:

def validate_referential_integrity(catalog):
    if 'namespaces' not in catalog:
        return  # Optional field, skip validation
    
    namespace_hashes = set(catalog['namespaces'].values())
    
    for diag_key in catalog['diags'].keys():
        if '-' in diag_key:
            ns_hash = diag_key.split('-')[0]
            if ns_hash not in namespace_hashes:
                warnings.warn(
                    f"Namespace hash '{ns_hash}' in key '{diag_key}' "
                    f"not found in namespaces index"
                )

Note: This is a SHOULD-level validation. Missing entries produce warnings, not errors, to support privacy-preserving catalogs.

6.2 Required Field Validation#

Structural Validation:

Valid JSON
Top-level is object
Required fields present (version, diags)
Compact identifiers match expected format (5 or 11 characters)
Diagnostic entries have required fields
Namespace fields follow constraints

Semantic Validation:

Version follows semver
CompactIDs are Base62 (5 characters)
CombinedIDs are Base62-Base62 (11 characters with hyphen)
Codes match WDP format
Severity matches code prefix
Field placeholders match fields list
Namespace fields consistent with key format

Example Validation Errors:

"Invalid JSON syntax at line 42"
"Missing required field: version"
"Missing required field: diags"
"Invalid compact ID length: 'ABC' (expected 5 or 11 characters)"
"Invalid severity: 'X' (must be E, B, C, W, H, S, K, I, or T)"
"Message placeholder {{user_id}} not in fields list"
"Cannot mix CompactID and CombinedID formats"
"Aggregated catalog cannot have top-level 'namespace' field"
"Single-namespace catalog cannot have 'namespaces' index"

6.3 Compatibility Checking#

Client Version Compatibility:

function isCompatible(catalogVersion, clientVersion) {
  const catMajor = parseInt(catalogVersion.split('.')[0]);
  const catMinor = parseInt(catalogVersion.split('.')[1]);
  const cliMajor = parseInt(clientVersion.split('.')[0]);
  const cliMinor = parseInt(clientVersion.split('.')[1]);
  
  // Major versions must match
  if (catMajor !== cliMajor) return false;
  
  // Client minor must be >= catalog minor
  return cliMinor >= catMinor;
}

// Examples:
isCompatible("1.2.0", "1.3.0") // true
isCompatible("1.3.0", "1.2.0") // false: client too old
isCompatible("2.0.0", "1.9.0") // false: major mismatch

7. Catalog Distribution#

7.1 Distribution Strategies#

7.1.1 Strategy 1: Static File Hosting (RECOMMENDED)#

Host catalog as static file on CDN:

https://cdn.example.com/catalogs/v1.0.0/catalog.json
https://cdn.example.com/catalogs/v1.0.0/catalog-compact.json
https://cdn.example.com/catalogs/v1.0.0/catalog-minimal.json

Benefits:

• Fast global distribution
• Built-in caching
• High availability
• Low cost

Best Practices:

• Use versioned URLs
• Set extended cache headers (365 days)
• Enable gzip/brotli compression
• Use HTTPS
• Enable CORS for browser clients

HTTP Headers:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Cache-Control: public, max-age=31536000, immutable
Content-Encoding: gzip
Access-Control-Allow-Origin: *
ETag: "abc123"

7.1.2 Strategy 2: API Endpoint#

Serve catalog via versioned API:

GET /api/v1/catalog
GET /api/v1/catalog?format=compact
GET /api/v1/catalog?version=1.2.0

Benefits:

• Dynamic catalog generation
• Access control (if required)
• Version negotiation

Best Practices:

• Cache responses aggressively
• Support conditional requests (ETag, If-None-Match)
• Allow format selection via query parameter
• Document API clearly

7.1.3 Strategy 3: Bundled with Application#

Bundle catalog directly in application:

/assets/catalog.json
/public/catalog.json

Benefits: No external dependency, immediate offline capability, guaranteed availability.

Drawbacks: Larger application bundle, requires app update to update catalog, no independent versioning.

Applicability: Small catalogs (<50KB), infrequent changes, offline-first apps.

7.2 Versioning Strategy#

7.2.1 Semantic Versioning#

Use semantic versioning for catalogs:

MAJOR.MINOR.PATCH

1.0.0 → Initial release
1.0.1 → Fix typo in message
1.1.0 → Add new diagnostics
2.0.0 → Breaking change (remove diagnostics, change format)

Rules:

• PATCH: Bug fixes, typo corrections, no new diagnostics
• MINOR: New diagnostics added, backward compatible
• MAJOR: Breaking changes (removed diagnostics, format changes)

7.2.2 Version in URL#

Include version in URL path:

https://cdn.example.com/catalogs/v{MAJOR}.{MINOR}.{PATCH}/catalog.json

Examples:

v1.0.0/catalog.json
v1.1.0/catalog.json
v2.0.0/catalog.json

Benefits: Immutable URLs, multiple versions can coexist, straightforward rollback.

7.3 Caching Recommendations#

7.3.1 Client-Side Caching#

Browser/Progressive Web App:

// Service worker caching
cache.put('catalog.json', response);

// LocalStorage for metadata
localStorage.setItem('catalog_version', '1.0.0');
localStorage.setItem('catalog_updated', Date.now());

Update Strategy:

1. Check current version
2. Fetch latest version metadata
3. If newer:
   a. Download in background
   b. Validate catalog
   c. Swap atomically
4. If same:
   a. Continue using cached version

Mobile App: Check for updates on startup (> 24h), on pull-to-refresh, or on network restore.

Cache Duration: Production: 7-30 days. Development: 1 hour.

7.3.2 Server-Side Caching#

CDN Configuration:

Cache-Control: public, max-age=31536000, immutable

Origin Server: Generate at build time, store in object storage, serve via CDN, never modify existing versions.

7.3.3 Compression#

Catalogs SHOULD be pre-compressed at build time. Zstandard is RECOMMENDED. Gzip and Brotli MAY be provided.

Recommended Compression:

Zstandard: 55-70% size reduction (fastest decompression)
Brotli:    55-65% size reduction (web-optimized)
Gzip:      50-60% size reduction (universal support)

Example Sizes:

catalog.json          150 KB
catalog.json.zst       45 KB (70% reduction)
catalog.json.br        50 KB (67% reduction)
catalog.json.gz        60 KB (60% reduction)

HTTP Configuration:

Accept-Encoding: zstd, br, gzip, deflate
Content-Encoding: zstd

Best Practice: Pre-compress at build time; serve compressed files directly. Generate .zst, .br, and .gz variants.

7.4 Aggregation Strategies#

7.4.1 Strategy 1: Build-Time Aggregation#

Combine multiple single-namespace catalogs into one aggregated catalog:

def aggregate_catalogs(catalogs_list):
    aggregated = {
        "version": "1.0.0",
        "namespaces": {},
        "diags": {}
    }
    
    for catalog in catalogs_list:
        namespace = catalog.get('namespace', 'unknown')
        namespace_hash = catalog.get('namespace_hash')
        
        if not namespace_hash:
            namespace_hash = generate_namespace_hash(namespace)
        
        aggregated['namespaces'][namespace] = namespace_hash
        
        for compact_id, diag in catalog['diags'].items():
            combined_id = f"{namespace_hash}-{compact_id}"
            aggregated['diags'][combined_id] = diag
    
    return aggregated

7.4.2 Strategy 2: Runtime Aggregation#

Gateway or tool loads multiple catalogs dynamically:

class CatalogAggregator {
  private catalogs: Map<string, Catalog> = new Map();
  
  async loadCatalog(namespace: string, url: string) {
    const catalog = await fetch(url).then(r => r.json());
    this.catalogs.set(namespace, catalog);
  }
  
  lookup(combinedId: string): DiagEntry | null {
    const [nsHash, compactId] = combinedId.split('-');
    
    for (const [ns, catalog] of this.catalogs) {
      if (catalog.namespace_hash === nsHash) {
        return catalog.diags[compactId];
      }
    }
    
    return null;
  }
}

7.4.3 Strategy 3: Manifest-Based#

Provide manifest file listing available catalogs:

{
  "version": "1.0.0",
  "catalogs": [
    {
      "namespace": "auth_service",
      "namespace_hash": "h4tYw2",
      "url": "/catalogs/auth_service-v1.0.0.json"
    },
    {
      "namespace": "payment_service",
      "namespace_hash": "k9Px3a",
      "url": "/catalogs/payment_service-v1.0.0.json"
    }
  ]
}

Appendix A: Complete Examples#

A.1 Full Format Example#

{
  "version": "1.0.0",
  "generated": "2024-01-15T10:30:00Z",
  "diags": {
    "jGKFp": {
      "code": "E.AUTH.TOKEN.001",
      "severity": "E",
      "message": "Token missing from Authorization header",
      "description": "The JWT token was not provided in the request. Authentication requires a valid Bearer token in the Authorization header.",
      "hints": [
        "Include header: Authorization: Bearer <token>",
        "Obtain token from /auth/login endpoint"
      ],
      "tags": ["auth", "security", "http"],
      "fields": []
    },
    "xY9Kp": {
      "code": "E.AUTH.TOKEN.EXPIRED",
      "severity": "E",
      "message": "Token expired at {{timestamp}}",
      "description": "The JWT token has exceeded its time-to-live (TTL). Please request a new token using your refresh token.",
      "hints": [
        "Use /auth/refresh endpoint with refresh token",
        "Check token expiration time (exp claim)"
      ],
      "tags": ["auth", "security", "jwt"],
      "fields": ["timestamp"]
    },
    "mN3Yr": {
      "code": "W.DATABASE.CONNECTION.027",
      "severity": "W",
      "message": "Database connection pool near capacity ({{current}}/{{max}})",
      "description": "The database connection pool is at 80% capacity. Performance may degrade if more connections are requested.",
      "hints": [
        "Monitor connection pool metrics",
        "Consider increasing pool size",
        "Review slow queries"
      ],
      "tags": ["database", "performance", "monitoring"],
      "fields": ["current", "max"]
    },
    "cP9Wm": {
      "code": "C.DISK.SPACE.CRITICAL",
      "severity": "C",
      "message": "Critical: Disk usage at {{usage}}% on {{mount_point}}",
      "description": "Disk space is critically low. System operations may fail if disk becomes full.",
      "hints": [
        "Free up disk space immediately",
        "Check for large log files",
        "Consider increasing disk capacity"
      ],
      "tags": ["disk", "critical", "infrastructure"],
      "fields": ["usage", "mount_point"]
    },
    "hK3Qn": {
      "code": "H.API.RATE.LIMIT",
      "severity": "H",
      "message": "Rate limit: {{remaining}} requests remaining in this window",
      "description": "This is informational - you are approaching your rate limit.",
      "hints": [
        "Implement exponential backoff",
        "Cache responses when possible",
        "Consider upgrading to higher tier"
      ],
      "tags": ["api", "rate-limit", "help"],
      "fields": ["remaining"]
    }
  }
}

A.2 Compact Format Example#

{
  "v": "1.0.0",
  "g": "2024-01-15T10:30:00Z",
  "wd": {
    "jGKFp": {
      "c": "E.AUTH.TOKEN.001",
      "s": "E",
      "m": "Token missing from Authorization header",
      "d": "The JWT token was not provided in the request.",
      "h": ["Include header: Authorization: Bearer <token>"],
      "t": ["auth", "security"],
      "f": []
    },
    "xY9Kp": {
      "c": "E.AUTH.TOKEN.EXPIRED",
      "s": "E",
      "m": "Token expired at {{timestamp}}",
      "d": "The JWT token has exceeded its TTL.",
      "h": ["Use /auth/refresh endpoint"],
      "t": ["auth", "jwt"],
      "f": ["timestamp"]
    },
    "mN3Yr": {
      "c": "W.DATABASE.CONNECTION.027",
      "s": "W",
      "m": "Database connection pool near capacity ({{current}}/{{max}})",
      "d": "The connection pool is at 80% capacity.",
      "h": ["Monitor connection pool metrics"],
      "t": ["database", "performance"],
      "f": ["current", "max"]
    },
    "cP9Wm": {
      "c": "C.DISK.SPACE.CRITICAL",
      "s": "C",
      "m": "Critical: Disk usage at {{usage}}% on {{mount_point}}",
      "d": "Disk space is critically low.",
      "h": ["Free up disk space immediately"],
      "t": ["disk", "critical"],
      "f": ["usage", "mount_point"]
    },
    "hK3Qn": {
      "c": "H.API.RATE.LIMIT",
      "s": "H",
      "m": "Rate limit: {{remaining}} requests remaining in this window",
      "d": "You are approaching your rate limit.",
      "h": ["Implement exponential backoff"],
      "t": ["api", "rate-limit"],
      "f": ["remaining"]
    }
  }
}

A.3 Minimal Format Example#

{
  "jGKFp": ["E.AUTH.TOKEN.001", "Token missing from Authorization header"],
  "xY9Kp": ["E.AUTH.TOKEN.EXPIRED", "Token expired at {{timestamp}}"],
  "mN3Yr": ["W.DATABASE.CONNECTION.027", "Database connection pool near capacity ({{current}}/{{max}})"],
  "cP9Wm": ["C.DISK.SPACE.CRITICAL", "Critical: Disk usage at {{usage}}% on {{mount_point}}"],
  "hK3Qn": ["H.API.RATE.LIMIT", "Rate limit: {{remaining}} requests remaining in this window"]
}

Appendix B: JSON Schema#

B.1 Full Format Schema#

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://waddling.dev/schemas/catalog-full.json",
  "title": "WDP Full Catalog Format",
  "type": "object",
  "required": ["version", "diags"],
  "additionalProperties": false,
  "properties": {
    "version": {
      "type": "string",
      "pattern": "^\\d+\\.\\d+\\.\\d+$"
    },
    "generated": {
      "type": "string",
      "format": "date-time"
    },
    "diags": {
      "type": "object",
      "patternProperties": {
        "^[0-9A-Za-z]{5}$": {
          "$ref": "#/definitions/diagEntry"
        }
      }
    }
  },
  "definitions": {
    "diagEntry": {
      "type": "object",
      "required": ["code", "severity", "message"],
      "properties": {
        "code": {
          "type": "string",
          "pattern": "^[EBCWHSKIT]\\.[A-Z]"
        },
        "severity": {
          "type": "string",
          "enum": ["E", "B", "C", "W", "H", "S", "K", "I", "T"]
        },
        "message": {
          "type": "string"
        },
        "description": {
          "type": "string"
        },
        "hints": {
          "type": "array",
          "items": { "type": "string" }
        },
        "tags": {
          "type": "array",
          "items": { "type": "string" }
        },
        "fields": {
          "type": "array",
          "items": { "type": "string" }
        }
      }
    }
  }
}

Appendix C: References#

C.1 Normative References#

C.2 Informative References#