Big Data Platform Engineer

Role Description

• You are operating Global Data Platform components (VM Servers, Kubernetes, Kafka) and applications (Apache Stack, Collibra, Databricks and similar).
• Implement automation of infrastructure, security components, and Continuous Integration & Continuous Delivery (CI/CD) for optimal execution of data pipelines (ELT/ETL).
• Develop solutions to build resiliency in data pipelines that perform health checks, monitoring, and alerting mechanisms; quality, timeliness, recency, and accuracy of the data delivery are improved.
• Apply DevSecOps and Agile approaches to deliver a holistic and integrated solution in iterative increments.
• Liaison and collaborate with enterprise security, digital engineering, and cloud operations architecture solution frameworks to drive consensus on design.
• Review system issues, incidents, and alerts to identify root causes and continuously implement features to improve platform performance.
• Be current on the latest industry developments and technology trends to effectively lead and design new features and capabilities.

Experience

• You have 5+ years of experience in building or designing large‑scale, fault‑tolerant, distributed systems.
• Migration experience of storage technologies (e.g. HDFS to S3 Object Storage).
• Integration of streaming and file‑based data ingestion/consumption (Kafka, Control‑M, AWS).
• Experience in DevOps, data pipeline development, and automation using Jenkins, Ansible, Chef, XL Release, and XL Deploy.
• Experience predominantly with on‑prem Big Data architecture; cloud migration experience is welcome.
• Hands‑on experience integrating Data Science Workbench platforms (e.g. Datiku).
• Experience with Agile project management methods (e.g. Scrum, SAFe).
• Supporting analytical value streams from enterprise reporting (e.g. Tableau) to data science (incl. ML Ops).

Skills

• Hands‑on working knowledge of large data solutions (e.g. data lakes, delta lakes, data meshes, data lakehouses, data platforms, data streaming solutions).
• In‑depth knowledge and experience in one or more large‑scale distributed technologies, including but not limited to:
• Hadoop ecosystem
• Kafka
• Kubernetes
• Spark
• Expert in Python and Java or another programming language (Scala/R, Linux/Unix scripting).
• VM setup and scaling (K8s scaling, managing Docker with Harbor, pushing images through CI/CD).
• Experience using data formats such as Apache Parquet, ORC, Avro.
• Exposure to machine learning algorithms is a plus.
• Good knowledge of German is beneficial; excellent command of English is essential.
• Knowledge of the financial sector and its products.