Skip to Content

Secrets Management

Overview

Earna AI uses a multi-layered secrets management strategy to securely store and distribute sensitive configuration data across development, staging, and production environments. This guide covers best practices, tooling, and critical deployment considerations.

Architecture

Google Secret Manager

Setup and Configuration

# Enable Secret Manager API gcloud services enable secretmanager.googleapis.com # Create secrets for different environments gcloud secrets create plaid-client-id --project=earna-production gcloud secrets create plaid-secret --project=earna-production gcloud secrets create token-encryption-key --project=earna-production gcloud secrets create supabase-service-key --project=earna-production gcloud secrets create temporal-encryption-key --project=earna-production

Version Management

# Add secret version echo -n "your_secret_value" | gcloud secrets versions add plaid-client-id --data-file=- # Access secret version gcloud secrets versions access latest --secret="plaid-client-id" # List all versions gcloud secrets versions list plaid-client-id

IAM Policies

# secret-manager-policy.yaml bindings: - members: - serviceAccount:temporal-worker@earna-production.iam.gserviceaccount.com - serviceAccount:plaid-sync@earna-production.iam.gserviceaccount.com role: roles/secretmanager.secretAccessor - members: - user:admin@earna.ai role: roles/secretmanager.admin
# Apply IAM policy gcloud secrets set-iam-policy plaid-client-id secret-manager-policy.yaml

GitHub Secrets

Organization-Level Secrets

Configure these secrets at the organization level for all repositories:

# Production secrets GOOGLE_CREDENTIALS # Service account key JSON GCP_PROJECT_ID # earna-production VERCEL_TOKEN # Vercel deployment token PLAID_CLIENT_ID # Production Plaid client ID PLAID_SECRET # Production Plaid secret TOKEN_ENCRYPTION_KEY # 32-byte hex encryption key SUPABASE_SERVICE_KEY # Production Supabase service key TEMPORAL_ENCRYPTION_KEY # Temporal data encryption key # Development secrets DEV_PLAID_CLIENT_ID # Sandbox Plaid client ID DEV_PLAID_SECRET # Sandbox Plaid secret DEV_SUPABASE_URL # Development Supabase URL DEV_SUPABASE_ANON_KEY # Development Supabase anon key

Repository-Specific Secrets

# Console-specific secrets NEXT_PUBLIC_SUPABASE_URL # Public Supabase URL NEXT_PUBLIC_SUPABASE_ANON_KEY # Public Supabase anon key NEXTAUTH_SECRET # NextAuth.js secret NEXTAUTH_URL # Authentication callback URL # API-specific secrets DATABASE_URL # Production database connection REDIS_URL # Redis connection string WEBHOOK_SECRET # Webhook verification secret

GitHub Actions Usage

# .github/workflows/deploy.yml name: Deploy to Production on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Google Cloud uses: google-github-actions/auth@v1 with: credentials_json: ${{ secrets.GOOGLE_CREDENTIALS }} - name: Get secrets from Secret Manager run: | echo "PLAID_CLIENT_ID=$(gcloud secrets versions access latest --secret=plaid-client-id)" >> $GITHUB_ENV echo "PLAID_SECRET=$(gcloud secrets versions access latest --secret=plaid-secret)" >> $GITHUB_ENV - name: Deploy to Vercel uses: amondnet/vercel-action@v25 with: vercel-token: ${{ secrets.VERCEL_TOKEN }} vercel-args: '--prod' working-directory: ./console

Vercel Environment Variables

Critical: CLI Usage Only

⚠️ NEVER use Vercel Dashboard UI for sensitive secrets - it introduces newlines that break encryption.

# Correct way - use CLI with printf printf "your_secret_here" | vercel env add TOKEN_ENCRYPTION_KEY production # Wrong way - Dashboard UI adds invisible newlines # ❌ Using Vercel Dashboard will break token decryption

Production Environment Setup

# Set production environment variables printf "$PLAID_CLIENT_ID" | vercel env add PLAID_CLIENT_ID production printf "$PLAID_SECRET" | vercel env add PLAID_SECRET production printf "$TOKEN_ENCRYPTION_KEY" | vercel env add TOKEN_ENCRYPTION_KEY production printf "$SUPABASE_SERVICE_KEY" | vercel env add SUPABASE_SERVICE_ROLE_KEY production # Set public environment variables (safe for UI) vercel env add NEXT_PUBLIC_PLAID_ENV production --value=production vercel env add NEXT_PUBLIC_SUPABASE_URL production --value=https://your-project.supabase.co

Environment Variable Validation

// lib/env-validation.ts import { z } from 'zod' const envSchema = z.object({ PLAID_CLIENT_ID: z.string().min(1, "Plaid client ID is required"), PLAID_SECRET: z.string().min(1, "Plaid secret is required"), TOKEN_ENCRYPTION_KEY: z.string().length(64, "Token encryption key must be 64 hex characters"), SUPABASE_SERVICE_ROLE_KEY: z.string().min(1, "Supabase service key is required"), }) export const env = envSchema.parse(process.env) // Validation helper for runtime checks export function validateEnvironmentVariables() { try { envSchema.parse(process.env) console.log('✅ Environment variables validated successfully') } catch (error) { console.error('❌ Environment validation failed:', error.errors) process.exit(1) } }

Automated Deployment Script

#!/bin/bash # scripts/deploy-production.sh set -e echo "🚀 Deploying to production..." # Validate required environment variables required_vars=( "PLAID_CLIENT_ID" "PLAID_SECRET" "TOKEN_ENCRYPTION_KEY" "SUPABASE_SERVICE_ROLE_KEY" "VERCEL_TOKEN" ) for var in "${required_vars[@]}"; do if [[ -z "${!var}" ]]; then echo "❌ Missing required environment variable: $var" exit 1 fi done # Set Vercel environment variables echo "📝 Setting Vercel environment variables..." printf "$PLAID_CLIENT_ID" | vercel env add PLAID_CLIENT_ID production --force printf "$PLAID_SECRET" | vercel env add PLAID_SECRET production --force printf "$TOKEN_ENCRYPTION_KEY" | vercel env add TOKEN_ENCRYPTION_KEY production --force printf "$SUPABASE_SERVICE_ROLE_KEY" | vercel env add SUPABASE_SERVICE_ROLE_KEY production --force # Deploy application echo "🚀 Deploying to Vercel..." vercel --prod --confirm echo "✅ Production deployment complete!"

Kubernetes Secrets

Creating Secrets from Secret Manager

# k8s/secret-manager-csi.yaml apiVersion: v1 kind: SecretProviderClass metadata: name: app-secrets namespace: platform-services spec: provider: gcp parameters: secrets: | - resourceName: "projects/earna-production/secrets/plaid-client-id/versions/latest" path: "plaid-client-id" - resourceName: "projects/earna-production/secrets/plaid-secret/versions/latest" path: "plaid-secret" - resourceName: "projects/earna-production/secrets/token-encryption-key/versions/latest" path: "token-encryption-key"

Temporal Worker Secret Mounting

# temporal/worker-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: temporal-worker namespace: temporal spec: template: spec: serviceAccountName: temporal-worker containers: - name: worker image: earna/temporal-worker:latest env: - name: PLAID_CLIENT_ID valueFrom: secretKeyRef: name: plaid-secrets key: client-id - name: PLAID_SECRET valueFrom: secretKeyRef: name: plaid-secrets key: secret - name: TOKEN_ENCRYPTION_KEY valueFrom: secretKeyRef: name: encryption-keys key: token-key volumeMounts: - name: secrets-store mountPath: "/mnt/secrets" readOnly: true volumes: - name: secrets-store csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "app-secrets"

Secret Rotation Script

#!/bin/bash # scripts/rotate-secrets.sh SECRET_NAME=$1 NEW_VALUE=$2 if [[ -z "$SECRET_NAME" || -z "$NEW_VALUE" ]]; then echo "Usage: $0 <secret-name> <new-value>" exit 1 fi echo "🔄 Rotating secret: $SECRET_NAME" # Add new version to Secret Manager echo -n "$NEW_VALUE" | gcloud secrets versions add "$SECRET_NAME" --data-file=- # Update Vercel environment printf "$NEW_VALUE" | vercel env add "$SECRET_NAME" production --force # Restart affected deployments kubectl rollout restart deployment/temporal-worker -n temporal kubectl rollout restart deployment/plaid-sync -n platform-services echo "✅ Secret rotation complete for: $SECRET_NAME"

Local Development Setup

.env.local Template

# .env.local template # Copy to .env.local and fill in actual values # Plaid Configuration PLAID_CLIENT_ID=your_sandbox_client_id PLAID_SECRET=your_sandbox_secret NEXT_PUBLIC_PLAID_ENV=sandbox PLAID_PRODUCTS=accounts,transactions,auth PLAID_COUNTRY_CODES=US,CA PLAID_WEBHOOK_URL=https://your-app.ngrok.io/api/plaid/webhook # Supabase Configuration NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co NEXT_PUBLIC_SUPABASE_ANON_KEY=your_anon_key SUPABASE_SERVICE_ROLE_KEY=your_service_role_key # Authentication NEXTAUTH_SECRET=your_nextauth_secret NEXTAUTH_URL=http://localhost:3000 # Encryption TOKEN_ENCRYPTION_KEY=64_character_hex_string_for_token_encryption_in_development_only # Database DATABASE_URL=postgresql://user:password@localhost:5432/earna_dev # Temporal TEMPORAL_ADDRESS=localhost:7233 TEMPORAL_NAMESPACE=default

Key Generation Script

#!/bin/bash # scripts/generate-keys.sh echo "🔐 Generating encryption keys..." # Generate 32-byte hex key for token encryption TOKEN_KEY=$(openssl rand -hex 32) echo "TOKEN_ENCRYPTION_KEY=$TOKEN_KEY" # Generate NextAuth secret NEXTAUTH_SECRET=$(openssl rand -base64 32) echo "NEXTAUTH_SECRET=$NEXTAUTH_SECRET" # Generate webhook secret WEBHOOK_SECRET=$(openssl rand -hex 32) echo "WEBHOOK_SECRET=$WEBHOOK_SECRET" echo "" echo "💾 Save these to your .env.local file"

Security Best Practices

Access Control

# secret-access-policy.yaml bindings: # Developers - read-only access to dev secrets - members: - group:developers@earna.ai role: roles/secretmanager.secretAccessor condition: title: "Dev secrets only" description: "Only access development secrets" expression: | resource.name.startsWith("projects/earna-production/secrets/dev-") # DevOps - full access to production secrets - members: - group:devops@earna.ai - serviceAccount:github-actions@earna-production.iam.gserviceaccount.com role: roles/secretmanager.admin condition: title: "Production secrets" description: "Full access to production secrets" expression: | !resource.name.contains("/secrets/dev-")

Audit Logging

# audit-config.yaml auditLogConfigs: - service: secretmanager.googleapis.com auditLogConfigs: - logType: DATA_READ - logType: DATA_WRITE - logType: ADMIN_READ

Secret Validation

// lib/secret-validation.ts export class SecretValidator { static validateTokenEncryptionKey(key: string): boolean { // Must be 64 hex characters (32 bytes) return /^[0-9a-fA-F]{64}$/.test(key) } static validatePlaidSecret(secret: string): boolean { // Plaid secrets follow specific format return secret.length >= 20 && !secret.includes('\n') } static validateSupabaseKey(key: string): boolean { // Supabase keys start with specific prefixes const validPrefixes = ['eyJ', 'sb-'] // JWT or Supabase format return validPrefixes.some(prefix => key.startsWith(prefix)) } static validateAll(secrets: Record<string, string>): ValidationResult { const errors: string[] = [] if (!this.validateTokenEncryptionKey(secrets.TOKEN_ENCRYPTION_KEY)) { errors.push('TOKEN_ENCRYPTION_KEY must be 64 hex characters') } if (!this.validatePlaidSecret(secrets.PLAID_SECRET)) { errors.push('PLAID_SECRET format is invalid') } if (!this.validateSupabaseKey(secrets.SUPABASE_SERVICE_ROLE_KEY)) { errors.push('SUPABASE_SERVICE_ROLE_KEY format is invalid') } return { valid: errors.length === 0, errors } } }

Troubleshooting

Common Issues

IssueCauseSolution
Token decryption failsNewlines in encryption keyUse CLI instead of Vercel UI
Secret not foundWrong project/environmentVerify project ID and secret name
Permission deniedMissing IAM rolesCheck service account permissions
Kubernetes mount failsWrong SecretProviderClassVerify CSI driver configuration

Diagnostic Commands

# Verify secret exists and is accessible gcloud secrets versions access latest --secret="token-encryption-key" # Check IAM permissions gcloud secrets get-iam-policy token-encryption-key # Verify Kubernetes secret mounting kubectl describe secretproviderclass app-secrets -n platform-services # Check pod secret access kubectl exec -it deployment/temporal-worker -n temporal -- env | grep PLAID

Emergency Secret Recovery

#!/bin/bash # scripts/emergency-secret-recovery.sh echo "🚨 Emergency secret recovery procedure" # Backup current secrets mkdir -p backups/$(date +%Y%m%d-%H%M%S) gcloud secrets list --format="value(name)" | while read secret; do echo "Backing up: $secret" gcloud secrets versions access latest --secret="$secret" > "backups/$(date +%Y%m%d-%H%M%S)/$(basename $secret)" done # Generate new encryption key NEW_TOKEN_KEY=$(openssl rand -hex 32) echo "Generated new TOKEN_ENCRYPTION_KEY: $NEW_TOKEN_KEY" # Rotate critical secrets ./scripts/rotate-secrets.sh token-encryption-key "$NEW_TOKEN_KEY" echo "✅ Emergency rotation complete"

Compliance and Auditing

Secret Access Monitoring

-- BigQuery query for secret access auditing SELECT timestamp, protoPayload.authenticationInfo.principalEmail, protoPayload.resourceName, protoPayload.methodName, protoPayload.requestMetadata.callerIp FROM `earna-production.audit_logs.cloudaudit_googleapis_com_data_access` WHERE protoPayload.serviceName = 'secretmanager.googleapis.com' AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR) ORDER BY timestamp DESC

Compliance Reporting

// lib/compliance-reporting.ts interface SecretAuditReport { secretName: string lastRotated: Date accessCount: number uniqueAccessors: string[] complianceStatus: 'compliant' | 'needs_rotation' | 'overdue' } export async function generateSecretAuditReport(): Promise<SecretAuditReport[]> { // Query secret access logs and generate compliance report // Implementation details... }

Contact Information

For secrets management issues:

Next Steps

  1. Environment Variables - Complete environment configuration
  2. GCP Setup - Google Cloud Platform configuration
  3. Deployment Guide - Production deployment procedures
  4. Monitoring - Infrastructure observability
Last updated on