NDPR-Compliant Field-Level Encryption in FastAPI: Securing BVN and NIN Storage with HashiCorp Vault & Blind Indexes

A mid-sized Lagos fintech handling agency banking received a formal audit notice from the Nigeria Data Protection Commission (NDPC). During an infrastructure review, security auditors discovered 420,000 Bank Verification Numbers (BVNs) and National Identification Numbers (NINs) stored in plaintext inside a read-replica PostgreSQL instance.

The engineering team assumed they were compliant because AWS KMS encryption at rest was enabled on the RDS volume. It was a costly misunderstanding. AWS TDE (Transparent Data Encryption) decrypts block storage beneath the database engine. If an attacker gets read access to PostgreSQL via a leaked database URI or SQL injection, TDE does nothing. The database happily serves plaintext PII. Under the Nigeria Data Protection Act (NDPA 2023), penalties reach up to ₦10,000,000 or 2% of annual gross revenue.

To comply with NDPR mandates, PII must be encrypted before it hits the database driver. However, naive application-level encryption breaks SQL lookups: if every BVN is encrypted with randomized AES-GCM tags, SELECT * FROM users WHERE bvn = '22123456789' fails because the ciphertext changes on every write.

Here is how to solve both problems in FastAPI using HashiCorp Vault's Transit Engine for envelope encryption and HMAC-SHA256 blind indexes for exact-match database queries.

Why Encryption-at-Rest Fails NDPA Security Audits

Database-level encryption at rest protects against physical disk theft from a data center. It does not protect against logical access breaches. When an application service connects to PostgreSQL, the database engine reads decrypted blocks from memory and returns plaintext columns.

Field-Level Encryption (FLE) shifts the cryptographic boundary back to the application layer. Before data leaves the Python runtime, sensitive values are encrypted into ciphertext blobs. PostgreSQL only sees random bytes.

+-----------------------------------------------------------------------+
|                           FastAPI Application                         |
|                                                                       |
|  Plaintext BVN ----> [ HMAC-SHA256 ] ------> Blind Index Hash        |
|        |                                           |                  |
|        +------------> [ Vault Transit API ] ----> AES-GCM Ciphertext   |
+----------------------------------------------------+------------------+
                                                     |                  |
                                                     v                  v
+-----------------------------------------------------------------------+
|                          PostgreSQL Database                          |
|                                                                       |
|  Table: users                                                         |
|  +---------------------------+-------------------------------------+  |
|  | bvn_bindex (BYTEA)        | bvn_encrypted (TEXT)                |  |
|  +---------------------------+-------------------------------------+  |
|  | \x8f2a11b9c0e7...        | vault:v1:8aF1kL...                  |  |
|  +---------------------------+-------------------------------------+  |
+-----------------------------------------------------------------------+

When capturing identity documents during user onboarding—often validated via integrations outlined in our Smile ID vs. Prembly vs. Youverify breakdown—you must treat the resulting identity tokens as high-risk assets.

FLE introduces a core challenge: standard search operations stop working. You cannot execute LIKE queries or equality filters on non-deterministic ciphertext. To fix this, we split the column into two separate entities:

1. An encrypted payload stored as randomized AES-256-GCM ciphertext.
2. A deterministic Blind Index computed via HMAC-SHA256 with a isolated secret key.

Setting Up HashiCorp Vault Transit Engine

Managing cryptographic keys inside application environment variables is a liability. If a worker process memory-dumps or logs its environment, your master key is leaked. We delegate encryption operations to HashiCorp Vault Transit Engine, using Vault as Encryption-as-a-Service.

First, enable the transit backend in your Vault cluster and create a key dedicated to user PII:

# Enable the transit secrets engine
vault secrets enable transit

# Create a key configured for AES-256-GCM encryption
vault write -f transit/keys/pii-bvn-key type=aes256-gcm96

# Configure key convergence (optional, keep false for non-deterministic FLE)
vault write transit/keys/pii-bvn-key/config deletion_allowed=false

Define an AppRole authentication policy for the FastAPI microservice so it can only encrypt and decrypt using this specific key:

path "transit/encrypt/pii-bvn-key" {
  capabilities = ["update"]
}
path "transit/decrypt/pii-bvn-key" {
  capabilities = ["update"]
}

Building Deterministic Blind Indexes with HMAC-SHA256

A blind index allows exact-match SQL queries without revealing the underlying plaintext. We pass the input string through HMAC-SHA256 using a high-entropy secret key ("pepper") stored separately from the database credentials.

Do not use plain SHA-256. Plain hashes are vulnerable to rainbow table attacks because the search space for 11-digit Nigerian BVNs is relatively small ($10^{11}$ possible permutations). HMAC-SHA256 with a 256-bit pepper completely neutralizes pre-computed lookup tables.

Here is our Python implementation using hashlib and hmac:

import hmac
import hashlib
import base64
from typing import Optional

class BlindIndexEngine:
    def __init__(self, pepper: str):
        if len(pepper) < 32:
            raise ValueError("Blind index pepper must be at least 32 characters long")
        self.pepper = pepper.encode("utf-8")

    def generate_index(self, value: Optional[str]) -> Optional[str]:
        """Generates a deterministic 32-byte HMAC string for database lookups."""
        if not value:
            return None
        
        # Normalize input: trim spaces, enforce lower/upper case rules
        clean_value = value.strip()
        
        # Compute HMAC-SHA256
        hashed = hmac.new(
            key=self.pepper,
            msg=clean_value.encode("utf-8"),
            digestmod=hashlib.sha256
        ).digest()
        
        # Return URL-safe base64 string for index storage
        return base64.urlsafe_b64encode(hashed).decode("utf-8")

Implementing the FastAPI & SQLAlchemy Cryptographic Type

Now we tie Vault Transit and the Blind Index together into a custom SQLAlchemy TypeDecorator. This transparently handles encryption on database writes and decryption on database reads, while syncing the blind index column automatically.

First, install hvac, the official Python client for HashiCorp Vault:

pip install hvac sqlalchemy pydantic-settings

Here is the full implementation of our service layer and database models:

import hvac
from sqlalchemy import String, TypeDecorator, Column, Integer, Event
from sqlalchemy.orm import declarative_base, Session
import os

Base = declarative_base()

class VaultService:
    def __init__(self):
        self.client = hvac.Client(
            url=os.getenv("VAULT_ADDR", "http://127.0.0.1:8200"),
            token=os.getenv("VAULT_TOKEN")
        )
        self.key_name = "pii-bvn-key"

    def encrypt(self, plaintext: str) -> str:
        if not plaintext:
            return None
        # Base64 encode the string before passing to Vault Transit
        import base64
        encoded_plaintext = base64.b64encode(plaintext.encode("utf-8")).decode("utf-8")
        
        response = self.client.secrets.transit.encrypt_data(
            name=self.key_name,
            plaintext=encoded_plaintext
        )
        return response["data"]["ciphertext"]

    def decrypt(self, ciphertext: str) -> str:
        if not ciphertext:
            return None
            
        response = self.client.secrets.transit.decrypt_data(
            name=self.key_name,
            ciphertext=ciphertext
        )
        import base64
        decoded_bytes = base64.b64decode(response["data"]["plaintext"])
        return decoded_bytes.decode("utf-8")

vault_service = VaultService()
blind_index_engine = BlindIndexEngine(pepper=os.getenv("BLIND_INDEX_PEPPER", "super-secret-pepper-string-at-least-32-chars"))

class EncryptedField(TypeDecorator):
    """SQLAlchemy type that encrypts data via Vault before saving to DB."""
    impl = String
    cache_ok = True

    def process_bind_param(self, value, dialect):
        if value is None:
            return None
        return vault_service.encrypt(value)

    def process_result_value(self, value, dialect):
        if value is None:
            return None
        return vault_service.decrypt(value)

class User(Base):
    __tablename__ = "users"

    id = Column(Integer, primary_key=True)
    email = Column(String(255), nullable=False)
    
    # Encrypted stored field (Vault Ciphertext string)
    bvn_encrypted = Column("bvn", EncryptedField(1024), nullable=True)
    
    # HMAC Blind Index for exact match querying
    bvn_bindex = Column(String(64), index=True, nullable=True)

    def set_bvn(self, raw_bvn: str):
        self.bvn_encrypted = raw_bvn
        self.bvn_bindex = blind_index_engine.generate_index(raw_bvn)

When querying for a user by BVN in FastAPI endpoints, query against the HMAC index, not the encrypted field:

from fastapi import FastAPI, Depends, HTTPException, status
from pydantic import BaseModel, BaseModel, Field

app = FastAPI(title="NDPR Compliant Service")

class UserOnboardSchema(BaseModel):
    email: str
    bvn: str = Field(..., min_length=11, max_length=11)

@app.post("/users/verify")
def get_user_by_bvn(payload: UserOnboardSchema, db: Session = Depends(get_db)):
    # Generate HMAC index from the incoming search query
    target_bindex = blind_index_engine.generate_index(payload.bvn)
    
    # Execute exact match search on indexed HMAC column
    user = db.query(User).filter(User.bvn_bindex == target_bindex).first()
    
    if not user:
        raise HTTPException(status_code=404, detail="User record not found")
        
    # Accessing user.bvn_encrypted automatically triggers Vault decryption
    return {
        "id": user.id,
        "email": user.email,
        "bvn": user.bvn_encrypted
    }

Refer to the OWASP Cryptographic Storage Cheat Sheet for secondary safeguards on key retention times and memory management.

Zero-Downtime Key Rotation Strategy

Under NDPR auditing standards, cryptographic keys should be rotated annually or immediately upon staff offboarding. Updating keys across tens of millions of records usually causes database lock timeouts or application downtime.

With Vault Transit, key rotation is executed server-side in Vault without immediately altering database records. Vault supports key versioning natively (`vault:v1:...