← Projects

OTP service and WAF hardening on AWS

Built a new one-time passcode service on AWS end to end, from Terraform to CI/CD, and hardened the WAF of a high-traffic consumer platform without blocking real users.

  • AWS
  • Terraform
  • Lambda
  • ElastiCache Redis
  • API Gateway
  • WAF
  • Kafka
  • GitHub Actions
  • Python
environments
3
environments
requests an hour behind the WAF
~900k
requests an hour behind the WAF
fraud signal types
11
fraud signal types
stored AWS keys in CI
0
stored AWS keys in CI

Infrastructure as code

  • Full Terraform stack for a new OTP service (create, verify and resend) across integration, UAT and production: container-based Lambda on arm64, ElastiCache Redis with TLS and auth (Multi-AZ in UAT), Secrets Manager, security groups, CloudWatch, and optional NAT egress switched on per environment. Existing VPCs were reused rather than duplicated.
  • Internal load balancer routing in the shared infrastructure repo: Lambda target groups, host-based listener rules and private Route 53 records, with isolated state per environment so nothing else is touched.
  • Moved the service from per-account ECR repos to a shared, cross-account registry, including cross-account pull permissions for Lambda and a cutover with no downtime.
  • Brought hand-made WAF web ACLs into Terraform by importing them, then added new rules as code.
  • Found that an old integration VPC was effectively dead and moved the workload to the live one.

Serverless

  • OTP Lambda shipped as a container image, with one handler that accepts both REST and HTTP API event formats.
  • Fixed Lambda rejecting multi-arch image manifests, and added smoke tests on every deploy.
  • Moved the API off the public internet to a private API Gateway endpoint, locked down with a resource policy.
  • Integrated with a managed Kafka cluster over VPC peering or NAT egress, depending on the environment.
  • WAF on Amplify-hosted frontends (CloudFront scope) for several web apps.

CI/CD

  • GitHub Actions pipeline that builds the image, pushes it to the central ECR, deploys and runs a smoke test.
  • Keyless pipeline auth through GitHub OIDC roles, with no stored AWS keys.
  • Feature work shipped as code, such as test-account bypass lists for the OTP service.

DevSecOps

  • Rate limiting on login, activation and account recovery pages, run in count mode first and then tuned.
  • AWS managed rule sets for IP reputation, common exploits and known bad inputs, Bot Control on the most-abused login endpoint, and Account Takeover Protection on a separate portal's login.
  • Admin access restricted to the corporate VPN with host-header and query-string WAF rules.
  • Audited the live WAF against the code and found unmanaged console changes and rules missing from code.
  • Every production change done safely: backups and reviewed dry-run diffs before each apply, lock-token and expected-state checks so nobody's console edits get overwritten, and before/after traffic comparisons to confirm no legitimate users were blocked.
  • Found and fixed a case-sensitivity mismatch that meant a fraud rule had never matched half its URLs, and rules sharing a CloudWatch metric name that made their numbers impossible to tell apart.
  • Kept UAT in step with production so WAF changes are tested before they ship.
  • Reviewed findings from an external security assessment, separating real gaps from ones already covered by a layer the assessors couldn't see.

Fraud detection and observability

  • Python tool that pulls 11 fraud signal types from application and WAF logs, including credential-stuffing patterns, suspicious IPs, activation velocity, card verification failures and device fingerprint spread.
  • Showed that the 10k-row cap in CloudWatch Logs Insights meant some queries could miss results, and proposed fixes: pagination, tighter filters and input from the data and fraud teams.
  • WAF logging, alarms and dashboards, including log-based dashboards where metrics weren't available.

Cost

  • Moved Bot Control to the end of the rule order and scoped it to specific endpoints, so the paid inspection runs on less traffic.
  • Flagged WAF log volume (about 2.8 TB stored) as a cost item, and weighed query cost and time in the fraud tooling.

Documentation

  • DevOps handover doc for the OTP service: environments, image and ECR, deploys, Terraform, routing, logs and access.
  • Evidence and change write-ups for every production change: before/after configs, test results and rollback steps.
← All projects