To Apply for this Job Click Here
Heitmeyer Consulting has a banking client that has a need within their Chief Data Office for a strong Data Engineer to specialize in building and optimizing high-performance, real-time data pipelines. This role is central to leveraging the power of Apache Kafka for event streaming and Apache Flink for complex, stateful stream processing and analytics. The ideal candidate will transform raw, high-velocity data into actionable, low-latency insights that drive core business functionality, working within our AWS-based data ecosystem leveraging S3 for storage. Role must be based in Dallas, TX or Tulsa, OK.
Top Required Skills:
- Require extensive experience as a Data Engineer building and maintaining production-grade data pipelines with a focus on real-time systems.
- In-depth experience with Apache Kafka along with hands-on experience with Apache Flink, including understanding of state management, fault tolerance and time semantics.
- Significant cloud background with strong expertise in AWS cloud services related to data engineering (S3, MSK, Glue, Athena, EMR, DynamoDB, etc.).
- Proficiency in programming/scripting languages such as Python, Java, or Scala.
- Familiar with modern data stack and big data technologies (Spark, Airflow, Iceberg, SQL) along with understanding of distributed systems and processing real-time data at scale.
- Excellent problem-solving skills and the ability to troubleshoot complex issues in distributed systems.
Nice-to-have:
- Background within financial services would be preferred but not required.
Key Responsibilities
- Design, develop, and deploy robust, high-throughput, low-latency streaming applications using Apache Flink and Java/Scala/Python APIs.
- Architect and optimize Kafka-based event-driven systems, including topic design, schema enforcement via Schema Registry, and integration using Kafka Connect.
- Lead data transformation through implementation of complex business logic, stateful processing, windowing, and data enrichment within Flink jobs to transform raw Kafka streams into analytics-ready datasets, ensuring exactly-once semantics and fault tolerance.
- Ensure cloud integration with storage and manage processed data efficiently in AWS S3 and potentially using an open format like Apache Iceberg, connecting seamlessly with downstream analytics tools (e.g., Athena).
- Optimize performance and observability – tune Flink and Kafka configurations for maximum performance, efficiency, and scalability, implementing robust monitoring and alerting for production environments.
- Partner closely with data scientists, software engineers, and business stakeholders to translate functional requirements into reliable data processing solutions.
- Champion software engineering best practices, including automated testing, code review, CI/CD pipelines, and documentation of data lineage and logic.
Heitmeyer Consulting is an equal opportunity employer, and we encourage all qualified candidates to apply. Qualified applicants will be considered without regard to minority status, gender, disability, veteran status or any other characteristic protected by law.