Secrets Management
Overview
Earna AI uses a multi-layered secrets management strategy to securely store and distribute sensitive configuration data across development, staging, and production environments. This guide covers best practices, tooling, and critical deployment considerations.
Architecture
Google Secret Manager
Setup and Configuration
# Enable Secret Manager API
gcloud services enable secretmanager.googleapis.com
# Create secrets for different environments
gcloud secrets create plaid-client-id --project=earna-production
gcloud secrets create plaid-secret --project=earna-production
gcloud secrets create token-encryption-key --project=earna-production
gcloud secrets create supabase-service-key --project=earna-production
gcloud secrets create temporal-encryption-key --project=earna-production
Version Management
# Add secret version
echo -n "your_secret_value" | gcloud secrets versions add plaid-client-id --data-file=-
# Access secret version
gcloud secrets versions access latest --secret="plaid-client-id"
# List all versions
gcloud secrets versions list plaid-client-id
IAM Policies
# secret-manager-policy.yaml
bindings:
- members:
- serviceAccount:temporal-worker@earna-production.iam.gserviceaccount.com
- serviceAccount:plaid-sync@earna-production.iam.gserviceaccount.com
role: roles/secretmanager.secretAccessor
- members:
- user:admin@earna.ai
role: roles/secretmanager.admin
# Apply IAM policy
gcloud secrets set-iam-policy plaid-client-id secret-manager-policy.yaml
GitHub Secrets
Organization-Level Secrets
Configure these secrets at the organization level for all repositories:
# Production secrets
GOOGLE_CREDENTIALS # Service account key JSON
GCP_PROJECT_ID # earna-production
VERCEL_TOKEN # Vercel deployment token
PLAID_CLIENT_ID # Production Plaid client ID
PLAID_SECRET # Production Plaid secret
TOKEN_ENCRYPTION_KEY # 32-byte hex encryption key
SUPABASE_SERVICE_KEY # Production Supabase service key
TEMPORAL_ENCRYPTION_KEY # Temporal data encryption key
# Development secrets
DEV_PLAID_CLIENT_ID # Sandbox Plaid client ID
DEV_PLAID_SECRET # Sandbox Plaid secret
DEV_SUPABASE_URL # Development Supabase URL
DEV_SUPABASE_ANON_KEY # Development Supabase anon key
Repository-Specific Secrets
# Console-specific secrets
NEXT_PUBLIC_SUPABASE_URL # Public Supabase URL
NEXT_PUBLIC_SUPABASE_ANON_KEY # Public Supabase anon key
NEXTAUTH_SECRET # NextAuth.js secret
NEXTAUTH_URL # Authentication callback URL
# API-specific secrets
DATABASE_URL # Production database connection
REDIS_URL # Redis connection string
WEBHOOK_SECRET # Webhook verification secret
GitHub Actions Usage
# .github/workflows/deploy.yml
name: Deploy to Production
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Google Cloud
uses: google-github-actions/auth@v1
with:
credentials_json: ${{ secrets.GOOGLE_CREDENTIALS }}
- name: Get secrets from Secret Manager
run: |
echo "PLAID_CLIENT_ID=$(gcloud secrets versions access latest --secret=plaid-client-id)" >> $GITHUB_ENV
echo "PLAID_SECRET=$(gcloud secrets versions access latest --secret=plaid-secret)" >> $GITHUB_ENV
- name: Deploy to Vercel
uses: amondnet/vercel-action@v25
with:
vercel-token: ${{ secrets.VERCEL_TOKEN }}
vercel-args: '--prod'
working-directory: ./console
Vercel Environment Variables
Critical: CLI Usage Only
⚠️ NEVER use Vercel Dashboard UI for sensitive secrets - it introduces newlines that break encryption.
# Correct way - use CLI with printf
printf "your_secret_here" | vercel env add TOKEN_ENCRYPTION_KEY production
# Wrong way - Dashboard UI adds invisible newlines
# ❌ Using Vercel Dashboard will break token decryption
Production Environment Setup
# Set production environment variables
printf "$PLAID_CLIENT_ID" | vercel env add PLAID_CLIENT_ID production
printf "$PLAID_SECRET" | vercel env add PLAID_SECRET production
printf "$TOKEN_ENCRYPTION_KEY" | vercel env add TOKEN_ENCRYPTION_KEY production
printf "$SUPABASE_SERVICE_KEY" | vercel env add SUPABASE_SERVICE_ROLE_KEY production
# Set public environment variables (safe for UI)
vercel env add NEXT_PUBLIC_PLAID_ENV production --value=production
vercel env add NEXT_PUBLIC_SUPABASE_URL production --value=https://your-project.supabase.co
Environment Variable Validation
// lib/env-validation.ts
import { z } from 'zod'
const envSchema = z.object({
PLAID_CLIENT_ID: z.string().min(1, "Plaid client ID is required"),
PLAID_SECRET: z.string().min(1, "Plaid secret is required"),
TOKEN_ENCRYPTION_KEY: z.string().length(64, "Token encryption key must be 64 hex characters"),
SUPABASE_SERVICE_ROLE_KEY: z.string().min(1, "Supabase service key is required"),
})
export const env = envSchema.parse(process.env)
// Validation helper for runtime checks
export function validateEnvironmentVariables() {
try {
envSchema.parse(process.env)
console.log('✅ Environment variables validated successfully')
} catch (error) {
console.error('❌ Environment validation failed:', error.errors)
process.exit(1)
}
}
Automated Deployment Script
#!/bin/bash
# scripts/deploy-production.sh
set -e
echo "🚀 Deploying to production..."
# Validate required environment variables
required_vars=(
"PLAID_CLIENT_ID"
"PLAID_SECRET"
"TOKEN_ENCRYPTION_KEY"
"SUPABASE_SERVICE_ROLE_KEY"
"VERCEL_TOKEN"
)
for var in "${required_vars[@]}"; do
if [[ -z "${!var}" ]]; then
echo "❌ Missing required environment variable: $var"
exit 1
fi
done
# Set Vercel environment variables
echo "📝 Setting Vercel environment variables..."
printf "$PLAID_CLIENT_ID" | vercel env add PLAID_CLIENT_ID production --force
printf "$PLAID_SECRET" | vercel env add PLAID_SECRET production --force
printf "$TOKEN_ENCRYPTION_KEY" | vercel env add TOKEN_ENCRYPTION_KEY production --force
printf "$SUPABASE_SERVICE_ROLE_KEY" | vercel env add SUPABASE_SERVICE_ROLE_KEY production --force
# Deploy application
echo "🚀 Deploying to Vercel..."
vercel --prod --confirm
echo "✅ Production deployment complete!"
Kubernetes Secrets
Creating Secrets from Secret Manager
# k8s/secret-manager-csi.yaml
apiVersion: v1
kind: SecretProviderClass
metadata:
name: app-secrets
namespace: platform-services
spec:
provider: gcp
parameters:
secrets: |
- resourceName: "projects/earna-production/secrets/plaid-client-id/versions/latest"
path: "plaid-client-id"
- resourceName: "projects/earna-production/secrets/plaid-secret/versions/latest"
path: "plaid-secret"
- resourceName: "projects/earna-production/secrets/token-encryption-key/versions/latest"
path: "token-encryption-key"
Temporal Worker Secret Mounting
# temporal/worker-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: temporal-worker
namespace: temporal
spec:
template:
spec:
serviceAccountName: temporal-worker
containers:
- name: worker
image: earna/temporal-worker:latest
env:
- name: PLAID_CLIENT_ID
valueFrom:
secretKeyRef:
name: plaid-secrets
key: client-id
- name: PLAID_SECRET
valueFrom:
secretKeyRef:
name: plaid-secrets
key: secret
- name: TOKEN_ENCRYPTION_KEY
valueFrom:
secretKeyRef:
name: encryption-keys
key: token-key
volumeMounts:
- name: secrets-store
mountPath: "/mnt/secrets"
readOnly: true
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "app-secrets"
Secret Rotation Script
#!/bin/bash
# scripts/rotate-secrets.sh
SECRET_NAME=$1
NEW_VALUE=$2
if [[ -z "$SECRET_NAME" || -z "$NEW_VALUE" ]]; then
echo "Usage: $0 <secret-name> <new-value>"
exit 1
fi
echo "🔄 Rotating secret: $SECRET_NAME"
# Add new version to Secret Manager
echo -n "$NEW_VALUE" | gcloud secrets versions add "$SECRET_NAME" --data-file=-
# Update Vercel environment
printf "$NEW_VALUE" | vercel env add "$SECRET_NAME" production --force
# Restart affected deployments
kubectl rollout restart deployment/temporal-worker -n temporal
kubectl rollout restart deployment/plaid-sync -n platform-services
echo "✅ Secret rotation complete for: $SECRET_NAME"
Local Development Setup
.env.local Template
# .env.local template
# Copy to .env.local and fill in actual values
# Plaid Configuration
PLAID_CLIENT_ID=your_sandbox_client_id
PLAID_SECRET=your_sandbox_secret
NEXT_PUBLIC_PLAID_ENV=sandbox
PLAID_PRODUCTS=accounts,transactions,auth
PLAID_COUNTRY_CODES=US,CA
PLAID_WEBHOOK_URL=https://your-app.ngrok.io/api/plaid/webhook
# Supabase Configuration
NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=your_anon_key
SUPABASE_SERVICE_ROLE_KEY=your_service_role_key
# Authentication
NEXTAUTH_SECRET=your_nextauth_secret
NEXTAUTH_URL=http://localhost:3000
# Encryption
TOKEN_ENCRYPTION_KEY=64_character_hex_string_for_token_encryption_in_development_only
# Database
DATABASE_URL=postgresql://user:password@localhost:5432/earna_dev
# Temporal
TEMPORAL_ADDRESS=localhost:7233
TEMPORAL_NAMESPACE=default
Key Generation Script
#!/bin/bash
# scripts/generate-keys.sh
echo "🔐 Generating encryption keys..."
# Generate 32-byte hex key for token encryption
TOKEN_KEY=$(openssl rand -hex 32)
echo "TOKEN_ENCRYPTION_KEY=$TOKEN_KEY"
# Generate NextAuth secret
NEXTAUTH_SECRET=$(openssl rand -base64 32)
echo "NEXTAUTH_SECRET=$NEXTAUTH_SECRET"
# Generate webhook secret
WEBHOOK_SECRET=$(openssl rand -hex 32)
echo "WEBHOOK_SECRET=$WEBHOOK_SECRET"
echo ""
echo "💾 Save these to your .env.local file"
Security Best Practices
Access Control
# secret-access-policy.yaml
bindings:
# Developers - read-only access to dev secrets
- members:
- group:developers@earna.ai
role: roles/secretmanager.secretAccessor
condition:
title: "Dev secrets only"
description: "Only access development secrets"
expression: |
resource.name.startsWith("projects/earna-production/secrets/dev-")
# DevOps - full access to production secrets
- members:
- group:devops@earna.ai
- serviceAccount:github-actions@earna-production.iam.gserviceaccount.com
role: roles/secretmanager.admin
condition:
title: "Production secrets"
description: "Full access to production secrets"
expression: |
!resource.name.contains("/secrets/dev-")
Audit Logging
# audit-config.yaml
auditLogConfigs:
- service: secretmanager.googleapis.com
auditLogConfigs:
- logType: DATA_READ
- logType: DATA_WRITE
- logType: ADMIN_READ
Secret Validation
// lib/secret-validation.ts
export class SecretValidator {
static validateTokenEncryptionKey(key: string): boolean {
// Must be 64 hex characters (32 bytes)
return /^[0-9a-fA-F]{64}$/.test(key)
}
static validatePlaidSecret(secret: string): boolean {
// Plaid secrets follow specific format
return secret.length >= 20 && !secret.includes('\n')
}
static validateSupabaseKey(key: string): boolean {
// Supabase keys start with specific prefixes
const validPrefixes = ['eyJ', 'sb-'] // JWT or Supabase format
return validPrefixes.some(prefix => key.startsWith(prefix))
}
static validateAll(secrets: Record<string, string>): ValidationResult {
const errors: string[] = []
if (!this.validateTokenEncryptionKey(secrets.TOKEN_ENCRYPTION_KEY)) {
errors.push('TOKEN_ENCRYPTION_KEY must be 64 hex characters')
}
if (!this.validatePlaidSecret(secrets.PLAID_SECRET)) {
errors.push('PLAID_SECRET format is invalid')
}
if (!this.validateSupabaseKey(secrets.SUPABASE_SERVICE_ROLE_KEY)) {
errors.push('SUPABASE_SERVICE_ROLE_KEY format is invalid')
}
return {
valid: errors.length === 0,
errors
}
}
}
Troubleshooting
Common Issues
Issue | Cause | Solution |
---|---|---|
Token decryption fails | Newlines in encryption key | Use CLI instead of Vercel UI |
Secret not found | Wrong project/environment | Verify project ID and secret name |
Permission denied | Missing IAM roles | Check service account permissions |
Kubernetes mount fails | Wrong SecretProviderClass | Verify CSI driver configuration |
Diagnostic Commands
# Verify secret exists and is accessible
gcloud secrets versions access latest --secret="token-encryption-key"
# Check IAM permissions
gcloud secrets get-iam-policy token-encryption-key
# Verify Kubernetes secret mounting
kubectl describe secretproviderclass app-secrets -n platform-services
# Check pod secret access
kubectl exec -it deployment/temporal-worker -n temporal -- env | grep PLAID
Emergency Secret Recovery
#!/bin/bash
# scripts/emergency-secret-recovery.sh
echo "🚨 Emergency secret recovery procedure"
# Backup current secrets
mkdir -p backups/$(date +%Y%m%d-%H%M%S)
gcloud secrets list --format="value(name)" | while read secret; do
echo "Backing up: $secret"
gcloud secrets versions access latest --secret="$secret" > "backups/$(date +%Y%m%d-%H%M%S)/$(basename $secret)"
done
# Generate new encryption key
NEW_TOKEN_KEY=$(openssl rand -hex 32)
echo "Generated new TOKEN_ENCRYPTION_KEY: $NEW_TOKEN_KEY"
# Rotate critical secrets
./scripts/rotate-secrets.sh token-encryption-key "$NEW_TOKEN_KEY"
echo "✅ Emergency rotation complete"
Compliance and Auditing
Secret Access Monitoring
-- BigQuery query for secret access auditing
SELECT
timestamp,
protoPayload.authenticationInfo.principalEmail,
protoPayload.resourceName,
protoPayload.methodName,
protoPayload.requestMetadata.callerIp
FROM `earna-production.audit_logs.cloudaudit_googleapis_com_data_access`
WHERE protoPayload.serviceName = 'secretmanager.googleapis.com'
AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
ORDER BY timestamp DESC
Compliance Reporting
// lib/compliance-reporting.ts
interface SecretAuditReport {
secretName: string
lastRotated: Date
accessCount: number
uniqueAccessors: string[]
complianceStatus: 'compliant' | 'needs_rotation' | 'overdue'
}
export async function generateSecretAuditReport(): Promise<SecretAuditReport[]> {
// Query secret access logs and generate compliance report
// Implementation details...
}
Contact Information
For secrets management issues:
- Email: devops@earna.ai
- Slack: #infrastructure
- Emergency: security-incident@earna.ai
Next Steps
- Environment Variables - Complete environment configuration
- GCP Setup - Google Cloud Platform configuration
- Deployment Guide - Production deployment procedures
- Monitoring - Infrastructure observability
Last updated on