Hire Freelance
Data Systems Scala Developers

Led the design and implementation of distributed data processing systems handling petabyte-scale datasets. Spearheaded the integration of Kafka with Spark for real-time analytics pipelines in financial services. Architected a high-performance data ingestion framework using Akka to optimize processing speed and reliability.

• Developed custom SQL query optimization techniques for large datasets
• Implemented Git versioning for complex data workflows
• Pioneered Akka-based microservices for scalable data processing
• Optimized Spark jobs reducing runtime by 30%

Developed large-scale distributed data processing systems for real-time analytics. Specialized in building high-performance ETL pipelines and data warehousing solutions. Designed and optimized custom consensus protocols for fault-tolerant systems.

• Implemented complex event-driven architectures for financial services.
• Architected scalable microservices using Go and Rust.
• Enhanced database internals for improved query performance.

Architected large-scale data pipelines using Apache Spark and Scala, optimizing for real-time analytics in the e-commerce domain. Spearheaded the migration of data infrastructure to Databricks, enhancing processing speed and reliability. Developed a custom data validation framework in Python, ensuring data integrity across distributed systems.

• Implemented Rust-based data transformation tools for performance gains
• Designed SQL-based ETL workflows for complex data models
• Automated data quality checks using custom-built Python scripts

Built a medallion-based data platform with Airflow and PySpark to standardize raw-material, production, supplier, and pricing data for margin analysis. Developed cross-entity data integration that mapped product relationships across heterogeneous source systems, enabling end-to-end product and cost traceability.

• Deployed data workloads across Azure and Kubernetes
• Modeled analytical data in PostgreSQL
• Instrumented pipelines and services with Grafana
• Refactored legacy Scala and TypeScript with expanded unit coverage

Designed MLS group-state features for secure messaging, including per-field permissions, in-place migration of live groups, and invite flows based on external commits. Built a Rust and Go bidirectional streaming transport with ordered delivery, heartbeats, reconnect handling, and database transaction boundaries enforced through async traits.

• Reduced 100-device invite payloads from 8 MB to 200 KB
• Cut full group loads from 13 database reads to one
• Built Rust-backed GraphQL and FFI layers for untrusted drivers
• Worked on edge computer-vision and blockchain-indexing systems

Replaced a costly external service with an in-house near-real-time platform processing 1.5 billion events daily, reducing latency from 15 minutes to 5. Designed custom watermark joins and dependency handling for late-arriving data, preserving unmatched records until upstream datasets were complete and correctly matched.

• Built Spark and Airflow production pipelines
• Operated workloads on Kubernetes and AWS
• Delivered audience data to roughly 30 partners
• Used Aerospike for durable intermediate state

Led the design and implementation of a distributed event streaming platform using Akka and Kafka. Spearheaded big data processing initiatives with Spark on AWS for analytics.

• Developed large-scale data pipelines with Apache Spark
• Managed team of engineers for complex system integrations
• Optimized server-side rendering with Play Framework

Designed and optimized large-scale data lakes and real-time streaming architectures using Apache Spark and Flink. Led migration of on-premise data warehouses to AWS cloud infrastructure, improving scalability and reducing latency. Specialized in developing ETL pipelines for financial analytics and risk management systems.

• Developed proprietary data ingestion framework for high-throughput processing
• Integrated machine learning models into data pipelines for predictive analytics
• Architected multi-region data replication solutions for disaster recovery

Specialized in building data pipelines using Scala and Java for large-scale ETL processes. Designed graph database solutions using Cypher and CQL to improve data retrieval efficiency.

• Automated data processing workflows with Groovy
• Enhanced data integration using Python scripts
• Implemented graph algorithms for advanced data insights

Engineered backend systems using Scala with a focus on functional programming. Built fault-tolerant services with ZIO for distributed systems. Developed GraphQL APIs using Caliban for seamless client-server interactions.

• Utilized Cats library for functional programming patterns
• Optimized SQL queries for high-volume transaction systems
• Developed Python utilities for backend service monitoring

Conducted research in flight data analysis, focusing on system and data analysis techniques. Designed experiments for data-driven insights and implemented solutions in Python and Matlab. Published findings in peer-reviewed journals.

• Researched flight data analysis
• Published in peer-reviewed journals
• Implemented solutions in Python and Matlab

Designed resilient backend systems utilizing Akka Persistence for stateful applications in e-commerce. Built real-time streaming solutions using Kafka Streams to process millions of transactions daily. Enhanced Spark applications for batch processing of large datasets in logistics.

• Engineered Akka Streams for low-latency data processing
• Developed Kafka-based event sourcing for audit trails
• Integrated Akka with third-party services for seamless data flow
• Optimized backend API performance for high-traffic environments

Developed real-time data processing systems utilizing Apache Kafka and Apache Flink to handle streaming data at scale. Designed and optimized ETL pipelines for large-scale data ingestion and transformation using Apache Spark. Implemented HBase solutions for high-throughput, low-latency data storage in big data environments.

• Architected distributed systems for high-volume data processing
• Built scalable data lakes with fault-tolerant architectures
• Optimized data transformation pipelines for performance and reliability

Builds high-throughput, low-latency distributed services for real-time bond trading, firehose-scale ingestion, regulated payments, and commerce. Has modernized legacy Java and Kotlin platforms into functional Scala microservices while operating Kafka- and Kinesis-driven systems on Kubernetes across AWS and GCP.

• Akka/Pekko Cluster and Persistence
• Back-pressure tuning in streaming pipelines
• Zero-downtime Akka-to-Pekko migration
• PostgreSQL production systems