Replication

Garak supports primary-secondary replication for high availability and disaster recovery. Secondary nodes stay synchronized with the primary through a combination of cold sync (full snapshots) and hot sync (incremental updates via oplog).

Architecture Overview

Primary Node                    Secondary Node(s)
    |                                  |
    |  1. Cold Sync (Initial)          |
    |---------------------------------->|
    |                                  |
    |  2. Hot Sync (Continuous)        |
    |---------------------------------->|
    |                                  |
    |  Writes → Oplog                  |
    |                                  |
    |  Oplog entries streamed          |
    |---------------------------------->|

Replication Modes

Cold Sync (Full Snapshot)

Cold sync creates a complete snapshot of the database and transfers it to secondary nodes.

When to use:

• Initial setup of a new secondary
• Secondary is too far behind (> 10 minutes)
• After catastrophic failure requiring full rebuild
• Scheduled full backups

Process:

1. Primary node creates a master lock
2. Database is tarred while locked
3. Lock is released, tar continues in background
4. Secondary downloads and extracts tar
5. Secondary updates its latest pointer
6. Secondary begins hot sync from snapshot time

Performance:

• Creates temporary disk pressure (tar file)
• Brief write lock during snapshot initiation
• Transfer time depends on database size and network

Hot Sync (Incremental Updates)

Hot sync tails the oplog to apply incremental changes.

When to use:

• Continuous replication
• Secondary is caught up or close behind
• Real-time data consistency

Process:

1. Secondary sends last known timestamp
2. Primary returns oplog entries since that time
3. Secondary applies entries in order
4. Secondary updates latest pointer
5. Process repeats every 1 second

Performance:

• Minimal overhead on primary
• Near real-time replication (1s polling)
• No locks required

Configuration

Primary Node Setup

Set these environment variables on your primary instance:

# Required
DATABASE_LOCATION=/path/to/database
PORT=1234
MASTER_TOKEN=your-secure-master-token

# Optional - for backups
AWS_S3_BACKUP_BUCKET=your-backup-bucket
AWS_REGION=us-east-1

Secondary Node Setup

Set these environment variables on your secondary instance:

# Required
DATABASE_LOCATION=/path/to/database
PORT=1234
MASTER_TOKEN=your-secure-master-token
GARAK_PRIMARY_URL=http://primary-host:1234

# Optional - for backups
AWS_S3_BACKUP_BUCKET=your-backup-bucket
AWS_REGION=us-east-1

Important

The MASTER_TOKEN must be identical on primary and all secondary nodes for authentication.

Secondary Node Behavior

When a secondary node starts:

1. Check for existing database:

   • If no top file exists → trigger cold sync
   • If database exists → check latest timestamp

2. Check if excessively behind:

   • If latest is > 10 minutes old → trigger cold sync
   • Backup existing data before cold sync
   • Delete old data after backup
   • Proceed with cold sync

3. Start hot sync thread:

   • Continuously poll primary every 1 second
   • Apply oplog entries incrementally
   • Update latest timestamp

Monitoring Replication

Check Replication Lag

On the secondary, check the latest file:

# Read the latest timestamp (u64 little-endian)
hexdump -e '8/1 "%02x"' /path/to/database/latest

Compare this timestamp with current time:

const now = Date.now();
const latest = /* read from file */;
const lagMs = now - latest;

console.log(`Replication lag: ${lagMs}ms`);

Check Oplog Size

Monitor the size of the oplog file to ensure it's not growing unbounded:

ls -lh /path/to/database/oplog

Cold Sync API

Initiate Cold Sync

curl -X POST http://primary:1234/replicas/cold_sync \
  -H "Authorization: master-token"

Response:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "accepted",
  "message": "Cold sync job started. Poll /replicas/cold_sync_status/{job_id} for status."
}

Check Job Status

curl http://primary:1234/replicas/cold_sync_status/{job_id} \
  -H "Authorization: master-token"

Response:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "Completed",
  "created_at": 1699900000000,
  "completed_at": 1699900120000,
  "tar_path": "/tmp/snapshot_123456.tar",
  "tar_size": 1048576000,
  "snapshot_time": 1699900000000
}

Status Values:

• "InProgress" - Snapshot still being created
• "Completed" - Ready to download
• {"Failed": "error message"} - Job failed

Download Snapshot

curl http://primary:1234/replicas/cold_sync_download/{job_id} \
  -H "Authorization: master-token" \
  -o snapshot.tar

Response Headers:

• Content-Type: application/x-tar
• x-garak-locktime: {snapshot_timestamp}

Hot Sync API

Request Oplog Entries

curl http://primary:1234/replicas/hot_sync \
  -H "Authorization: master-token" \
  -H "x-garak-latest: 1699900000000"

Request Headers:

• x-garak-latest: Last known timestamp from secondary

Response:

[
  {
    "Write": {
      "silo": "my-silo",
      "shard": "shard-1",
      "store": "users",
      "node_id": 12345,
      "data": { "email": "user@example.com", "name": "User" },
      "is_new_node": true
    }
  },
  {
    "Delete": {
      "silo": "my-silo",
      "shard": "shard-1",
      "store": "users",
      "node_id": 67890
    }
  }
]

Response Headers:

• x-garak-latest: Current timestamp on primary

Oplog Format

The oplog is a line-delimited file at {DATABASE_LOCATION}/oplog:

{timestamp};{json_entry}
{timestamp};{json_entry}
...

Example:

1699900000000;{"Write":{"silo":"my-silo","shard":"shard-1","store":"users","node_id":123,"data":{"name":"Alice"},"is_new_node":true}}
1699900001000;{"Delete":{"silo":"my-silo","shard":"shard-1","store":"users","node_id":456}}

Entry Types:

Write:

{
  "Write": {
    "silo": "string",
    "shard": "string",
    "store": "string",
    "node_id": number,
    "data": {...},
    "is_new_node": boolean
  }
}

Delete:

{
  "Delete": {
    "silo": "string",
    "shard": "string",
    "store": "string",
    "node_id": number
  }
}

Failure Scenarios

Primary Node Failure

Secondary nodes cannot become primaries automatically. Manual intervention required:

1. Stop all secondaries to prevent conflicts
2. Promote one secondary:
   • Remove GARAK_PRIMARY_URL from environment
   • Restart as new primary
3. Update other secondaries to point to new primary
4. Update application to use new primary URL

Secondary Node Failure

1. Secondary restarts automatically
2. Checks replication lag
3. If < 10 minutes behind: resumes hot sync
4. If > 10 minutes behind: triggers cold sync

Network Partition

If secondary loses connection to primary:

• Hot sync requests will fail
• Secondary continues serving read-only traffic (stale data)
• When connection resumes:
  • If lag < 10 minutes: resume hot sync
  • If lag > 10 minutes: trigger cold sync

Option 2: Trigger cold sync on secondaries

All secondaries can rebuild from fresh snapshot, then old oplog can be truncated.

Backup Integration

Garak includes automatic S3 backup functionality:

# Configure S3 backups
export AWS_S3_BACKUP_BUCKET=my-backup-bucket
export AWS_REGION=us-east-1

Automatic Behaviors:

• Backup created before risky operations (cold sync on existing data)
• Periodic backups via background thread
• Uses AWS SDK with default credential chain

Best Practices

High Availability Setup

1. Run at least 2 secondaries for redundancy
2. Distribute secondaries across availability zones
3. Monitor replication lag with alerting
4. Test failover procedures regularly
5. Keep oplog size manageable (rotate or trigger cold syncs)

Performance Optimization

1. Dedicated network for replication traffic
2. SSD storage on all nodes
3. Monitor disk I/O during cold sync
4. Limit concurrent cold syncs to avoid primary overload

Disaster Recovery

1. Regular S3 backups (daily or more frequent)
2. Test restore procedures from backups
3. Document failover process for team
4. Monitor backup job success with alerting

Operational Considerations

1. Plan cold sync during low traffic periods
2. Verify secondary data after cold sync
3. Keep master token secure and rotated
4. Monitor secondary disk space (tar files require temporary space)
5. Use consistent DATABASE_LOCATION across nodes

Troubleshooting

Secondary Not Catching Up

Symptoms:

• Replication lag increasing
• Hot sync retrieving entries but lag not decreasing

Solutions:

1. Check network connectivity between nodes
2. Verify master token matches
3. Check secondary disk space
4. Trigger manual cold sync

Cold Sync Failing

Symptoms:

• Job status shows Failed
• Download returns 500 error

Solutions:

1. Check primary disk space for tar file
2. Verify master lock is not stuck
3. Check primary logs for errors
4. Retry with new job

Oplog Growing Rapidly

Symptoms:

• Oplog file size continuously increasing
• Disk space depleting

Solutions:

1. Check for write-heavy workload (expected behavior)
2. Ensure secondaries are consuming oplog
3. Consider oplog rotation during maintenance window
4. Add more secondaries to distribute load

Replication Lag Oscillating

Symptoms:

• Lag increases, triggers cold sync
• Completes, but lag increases again quickly

Solutions:

1. Secondary may be under-provisioned (CPU/disk)
2. Check for slow network between nodes
3. Verify no resource contention on secondary
4. Consider increasing hot sync frequency (requires code change)