Handling Out-of-Order Paystack Webhooks: Building an Idempotent Wallet Engine with Redis and PostgreSQL

It is Friday evening at 8:00 PM in Lagos. Your mobile app users are funding their digital wallets via bank transfers and cards. Paystack receives a settlement notification from NIBSS, updates the transaction status, and dispatches a charge.success webhook payload to your servers. However, a transient routing bottleneck between your edge load balancer and your application workers causes a 45-second TCP socket timeout. Paystack’s webhooks engine receives no HTTP status response, marks the delivery attempt as failed, and enqueues an automated backoff retry.

Unaware of the network layer timeout, the end-user stays on your app screen, sees the loading wheel freeze, and hits the manual "I've Paid" trigger. This fires off a synchronous client-side verification call (GET /transaction/verify/:reference) to your application backend.

Your API receives the verification request, queries Paystack's REST endpoint, receives a status: true response, credits ₦50,000 to the user's PostgreSQL wallet balance, and appends a row to your ledger table. Thirty seconds later, Paystack's original, delayed webhook payload finally arrives, followed almost instantly by the automated retry attempt from Paystack’s delivery queue.

Because your webhook worker process reads the database state without atomic locking, two worker threads process these concurrent webhook deliveries simultaneously. Both check the database, find no record of the specific webhook event ID in your audit logs, and execute a balance credit operation. By 8:05 PM, the user's account has been credited ₦150,000 instead of ₦50,000. Before your batch reconciliation script runs at midnight, the user has moved ₦150,000 out to an external GTBank account, leaving your platform with an irrecoverable ₦100,000 balance deficit.

This exact failure mode plays out across Nigerian fintech platforms every single week. Flaky mobile networks, aggressive client retry logic, and provider retry mechanisms combine to make out-of-order and duplicate webhook deliveries an absolute guarantee, not an edge case.

To safeguard your platform against financial losses, you must construct a deterministic event consumer using signature validation, distributed Redis locks, and double-entry database transactions in PostgreSQL.

---

HMAC Signature Verification and Signature Replay Prevention

Before executing business logic on an incoming webhook payload, you must confirm that the payload was generated by Paystack and was not modified in transit. Paystack signs all outgoing HTTP POST requests with a SHA-512 HMAC digest placed in the x-paystack-signature HTTP header. The secret key used to compute this hash is your account's private Paystack API key.

Never parse the body payload with your application router prior to calculating the hash. Parsing JSON bodies often standardizes whitespace or reorders object key key-value pairs, which mutates the raw byte array and leads to false HMAC calculation failures.

Here is how to extract and validate the raw byte stream using Node.js and Express middleware:

import crypto from 'node:crypto';
import { Request, Response, NextFunction } from 'express';

export interface AuthenticatedWebhookRequest extends Request {
  rawBody?: Buffer;
}

export const verifyPaystackSignature = (
  secretKey: string
) => {
  return (req: AuthenticatedWebhookRequest, res: Response, next: NextFunction) => {
    const signature = req.headers['x-paystack-signature'] as string;

    if (!signature) {
      return res.status(401).json({ error: 'Missing Paystack signature header' });
    }

    // Raw body must be captured as a Buffer before JSON parsing occurs
    const rawBody = req.rawBody;
    if (!rawBody) {
      return res.status(500).json({ error: 'Server misconfiguration: Raw body not captured' });
    }

    const hash = crypto
      .createHmac('sha512', secretKey)
      .update(rawBody)
      .digest('hex');

    // Use timingSafeEqual to protect against side-channel timing attacks
    const signatureBuffer = Buffer.from(signature, 'utf8');
    const hashBuffer = Buffer.from(hash, 'utf8');

    if (signatureBuffer.length !== hashBuffer.length || 
        !crypto.timingSafeEqual(signatureBuffer, hashBuffer)) {
      return res.status(400).json({ error: 'Invalid HMAC signature signature' });
    }

    return next();
  };
};

For additional context on securing API routes against automated scanning and replay exploits, review our analysis on Defending Against BVN Enumeration and NIP Replay Fraud: Hardening a 4M-User Wallet Engine.

---

Distributed Idempotency and Race Condition Locking with Redis

Once a payload passes signature validation, you must enforce execution idempotency. Webhook payloads delivered concurrently by multiple worker pods will pass simple database queries like SELECT * FROM processed_events WHERE reference = 'x' if those read operations run before any thread completes its write transaction.

To lock incoming reference IDs globally across distributed application nodes, use Redis atomic keys with an explicit time-to-live (TTL). The SET key value NX EX seconds command guarantees that exactly one process acquires the processing lock for a transaction reference at any given instant.

Below is an implementation of a Redis distributed lock mechanism built using ioredis:

import Redis from 'ioredis';

export class DistributedLockService {
  constructor(private readonly redis: Redis) {}

  /**
   * Attempts to acquire an atomic processing lock for a transaction reference.
   * @param reference Unique payment reference from Paystack
   * @param ttlSeconds Lock expiry time to prevent deadlocks on worker crashes
   */
  async acquireLock(reference: string, ttlSeconds: number = 30): Promise<boolean> {
    const lockKey = `lock:webhook:${reference}`;
    const result = await this.redis.set(lockKey, 'processing', 'EX', ttlSeconds, 'NX');
    return result === 'OK';
  }

  /**
   * Explicitly releases the lock after successful processing.
   */
  async releaseLock(reference: string): Promise<void> {
    const lockKey = `lock:webhook:${reference}`;
    await this.redis.del(lockKey);
  }

  /**
   * Check if an event ID has already been fully processed and committed.
   */
  async isEventProcessed(eventId: number): Promise<boolean> {
    const eventKey = `processed:event:${eventId}`;
    const exists = await this.redis.exists(eventKey);
    return exists === 1;
  }

  /**
   * Cache processed event IDs for 72 hours to short-circuit repeated retries.
   */
  async markEventProcessed(eventId: number): Promise<void> {
    const eventKey = `processed:event:${eventId}`;
    await this.redis.set(eventKey, '1', 'EX', 259200);
  }
}

Refer to the official Paystack Webhooks Documentation for additional details on their event delivery retries and backoff schedules.

---

Implementing a Strict Wallet Ledger State Machine in PostgreSQL

Distributed locks prevent overlapping executions, but they cannot solve out-of-order processing on their own. For example, if a transfer.reversed event arrives before a delayed transfer.success event due to network routing anomalies, processing the reversal first will cause your ledger engine to register an invalid balance decrement on an uncredited wallet.

You must model payment references using an explicit database state machine and enforce state transitions inside PostgreSQL transactions using SELECT ... FOR UPDATE row locks.

Database Schema Setup

CREATE TYPE transaction_status AS ENUM ('PENDING', 'SUCCESSFUL', 'FAILED', 'REVERSED');
CREATE TYPE ledger_entry_type AS ENUM ('CREDIT', 'DEBIT');

CREATE TABLE wallets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL UNIQUE,
    balance_kobo BIGINT NOT NULL DEFAULT 0 CHECK (balance_kobo >= 0),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE payment_requests (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    reference VARCHAR(128) NOT NULL UNIQUE,
    wallet_id UUID NOT NULL REFERENCES wallets(id),
    amount_kobo BIGINT NOT NULL CHECK (amount_kobo > 0),
    status transaction_status NOT NULL DEFAULT 'PENDING',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE ledger_entries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    wallet_id UUID NOT NULL REFERENCES wallets(id),
    payment_request_id UUID NOT NULL REFERENCES payment_requests(id),
    entry_type ledger_entry_type NOT NULL,
    amount_kobo BIGINT NOT NULL CHECK (amount_kobo > 0),
    balance_after_kobo BIGINT NOT NULL CHECK (balance_after_kobo >= 0),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE processed_webhook_events (
    event_id BIGINT PRIMARY KEY,
    event_type VARCHAR(64) NOT NULL,
    reference VARCHAR(128) NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Transactional Service Implementation

To ensure complete database safety, wrap the wallet balance updates inside explicit PostgreSQL isolation blocks. See the PostgreSQL Explicit Locking Documentation for detailed behavioral mechanics regarding FOR UPDATE lock acquisitions.

import { Pool, PoolClient } from 'pg';

interface PaystackEventPayload {
  id: number;
  event: string;
  data: {
    reference: string;
    amount: number; // Paystack sends amounts in kobo
    status: string;
    [key: string]: any;
  };
}

export class WalletLedgerEngine {
  constructor(
    private readonly dbPool: Pool,
    private readonly lockService: DistributedLockService
  ) {}

  async processChargeSuccess(payload: PaystackEventPayload): Promise<{ status: string }> {
    const { id: eventId, event, data } = payload;
    const { reference, amount: amountKobo } = data;

    // Step 1: Check quick Redis cache for idempotency
    const alreadyProcessed = await this.lockService.isEventProcessed(eventId);
    if (alreadyProcessed) {
      return { status: 'ALREADY_PROCESSED' };
    }

    // Step 2: Acquire short-lived Redis distributed lock for reference
    const lockAcquired = await this.lockService.acquireLock(reference);
    if (!lockAcquired) {
      // Return an error code that signals the express route to return HTTP 429 or 200 based on policy
      throw new Error('CONCURRENT_PROCESSING_IN_PROGRESS');
    }

    const client: PoolClient = await this.dbPool.connect();

    try {
      await client.query('BEGIN;');

      // Step 3: Check PostgreSQL duplicate audit table
      const eventCheck = await client.query(
        'SELECT event_id FROM processed_webhook_events WHERE event_id = $1 FOR UPDATE;',
        [eventId]
      );

      if (eventCheck.rowCount && eventCheck.rowCount > 0) {
        await client.query('ROLLBACK;');
        await this.lockService.markEventProcessed(eventId);
        return { status: 'ALREADY_PROCESSED' };
      }

      // Step 4: Lock the payment request record to evaluate state transition
      const paymentRes = await client.query(
        'SELECT id, wallet_id, amount_kobo, status FROM payment_requests WHERE reference = $1 FOR UPDATE;',
        [reference]
      );

      if (paymentRes.rowCount === 0) {
        // Payment request record doesn't exist locally; abort and rollback
        await client.query('ROLLBACK;');
        throw new Error(`UNRECOGNIZED_REFERENCE: ${reference}`);
      }

      const payment = paymentRes.rows[0];

      // Step 5: Enforce strict state machine transitions
      if (payment.status === 'SUCCESSFUL') {
        // Already processed via synchronous verification flow
        await client.query(
          'INSERT INTO processed_webhook_events (event_id, event_type, reference) VALUES ($1, $2, $3);',
          [eventId, event, reference]
        );
        await client.query('COMMIT;');
        await this.lockService.markEventProcessed(eventId);
        return { status: 'STATE_ALREADY_SET' };
      }

      if (payment.status !== 'PENDING') {
        await client.query('ROLLBACK;');
        throw new Error(`INVALID_STATE_TRANSITION from ${payment.status} to SUCCESSFUL`);
      }

      // Step 6: Lock the target wallet balance row exclusively
      const walletRes = await client.query(
        'SELECT balance_kobo FROM wallets WHERE id = $1 FOR UPDATE;',
        [payment.wallet_id]
      );

      const currentBalance = BigInt(walletRes.rows[0].balance_kobo);
      const creditAmount = BigInt(amountKobo);
      const newBalance = currentBalance + creditAmount;

      // Step 7: Update wallet balance
      await client.query(
        'UPDATE wallets SET balance_kobo = $1, updated_at = NOW() WHERE id = $2;',
        [newBalance.toString(), payment.wallet_id]
      );

      // Step 8: Create ledger audit entry
      await client.query(
        `INSERT INTO ledger_entries 
         (wallet_id, payment_request_id, entry_type, amount_kobo, balance_after_kobo) 
         VALUES ($1, $2, $3, $4, $5);`,
        [payment.wallet_id, payment.id, 'CREDIT', creditAmount.toString(), newBalance.toString()]
      );

      // Step 9: Update payment status
      await client.query(
        'UPDATE payment_requests SET status = $1, updated_at = NOW() WHERE id = $2;',
        ['SUCCESSFUL', payment.id]
      );

      // Step 10: Record event in idempotency log
      await client.query(
        'INSERT INTO processed_webhook_events (event_id, event_type, reference) VALUES ($1, $2, $3);',
        [eventId, event, reference]
      );

      await client.query('COMMIT;');
      await this.lockService.markEventProcessed(eventId);

      return { status: 'SUCCESSFULLY_MUTATED' };
    } catch (err) {
      await client.query('ROLLBACK;');
      throw err;
    } finally {
      client.release();
      await this.lockService.releaseLock(reference);
    }
  }
}

For teams building international checkout engines that handle multiple currencies alongside standard Paystack billing, read our guide on Dual-Currency SaaS Billing in West Africa: Implementing Real-Time FX Pegging and Metered Subscriptions with Paystack and Stripe.

---

Handling Delayed Out-of-Order Webhooks vs Polling Operations

In high-volume fintech engines, you cannot rely entirely on webhooks. Network providers in Nigeria suffer periodic routing degradation, which causes webhooks to drop or stall for hours. To ensure fast order settlement, your engine must use background reconciliation jobs that poll the Paystack API for unconfirmed transactions.

However, when a background polling worker and a delayed webhook processor attempt to update the same reference simultaneously, uncoordinated execution can cause data corruption. To resolve this, enforce these architectural rules:

1. Unified Mutation Vector: Route both webhook events and API polling results through the exact same ledger execution code path (processChargeSuccess). Never write separate update queries for polling handlers.
2. Database Isolation Levels: Ensure your connection pool executes mutating transactions using READ COMMITTED or REPEATABLE READ levels, combined with row-level FOR UPDATE locks as demonstrated in the code above.
3. Integer Money Representation: Store all local balances and transaction amounts in Kobo (integers). Never use floating-point types (FLOAT, DOUBLE) in JavaScript or SQL math operations. Floating-point operations introduce subtle precision errors that cause balance check constraints to throw false assertion errors.

| Handling Layer | Webhook Input | Polling Fallback Input | | :--- | :--- | :--- | | Trigger Point | Incoming HTTP POST from Paystack | Cron Job / Queue Worker (e.g., every 5 mins) | | Concurrency Strategy | Redis SET NX key on reference | Redis SET NX key on reference | | Database Lock | SELECT ... FOR UPDATE on wallet_id | SELECT ... FOR UPDATE on wallet_id | | State Transition | PENDING -> SUCCESSFUL | PENDING -> SUCCESSFUL | | Duplicate Result | Skips execution & returns HTTP 200 | Skips execution & records job completion |

---

Common Pitfalls in Nigerian Webhook Integrations

1. Returning HTTP 500 Responses for Duplicate Events

When your system detects an event payload that has already been processed, do not return an HTTP 500 error response. Paystack treats 500 status codes as processing failures and will keep retrying delivery for up to 72 hours. This pollutes your error monitoring tools (e.g., Sentry) and risks temporary API suspension. Always log the duplicate gracefully and return an HTTP 200 OK status.

2. Performing Synchronous Network Requests Inside Database Transactions

Never execute outbound HTTPS requests (such as calling Paystack's /transaction/verify endpoint) inside an active PostgreSQL transaction block (BEGIN ... COMMIT). If the network call latency spikes to 10 seconds, that database connection remains locked open. Under heavy load, this exhausts your connection pool, spikes memory consumption, and drops all incoming user traffic.

3. Relying Solely on JavaScript In-Memory Locks

Using Node.js in-memory structures like Set or Map to track active reference locks works fine on a single development machine. However, production deployments run multiple app instances across containers or serverless functions. Node.js processes cannot share in-memory state across instances. Use a shared memory store like Redis to ensure locks are globally accessible across all workers.

---

Webhook Handling FAQ

Why does Paystack send duplicate webhooks even after my server returned an HTTP 200 OK status?

Network timeouts can occur on the return path. If your server receives the payload and writes a 200 OK response, but an intermediate edge proxy or network operator drops the returning TCP packet before it reaches Paystack, Paystack's client flags the delivery as timed out. Paystack will then enqueue an automated retry. Your system must treat duplicate event deliveries as expected behavior.

Should I verify transactions synchronously via API or rely exclusively on webhooks?

Use both. Webhooks offer fast, real-time status updates without hitting API rate limits. However, because network glitches can delay webhooks, supplement them with a background polling cron job that checks transactions that remain in a PENDING state after 15 minutes.

How long should I keep idempotency keys in Redis?

Keep idempotency keys in Redis for at least 72 hours. Paystack retries failed webhook deliveries across a 72-hour window. Maintaining these keys in Redis ensures retries are checked quickly without putting excess query load on your primary PostgreSQL database.

---

Final Implementation Checklist

To ensure your ledger engine is completely safe against duplicate credits, verify that your implementation satisfies these four requirements:

• Extract the raw request body as an unparsed Buffer before passing it to signature verification middleware.
• Use crypto.timingSafeEqual during HMAC comparison to prevent timing attacks.
• Acquire a shared Redis key lock (SET NX EX) on the payment reference before beginning database transactions.
• Execute balance mutations inside PostgreSQL transactions using SELECT FOR UPDATE locks on wallet rows, and maintain integer amounts (kobo) throughout your application pipeline.