Senior Data Engineer

• Data Pipeline Development for Analytics:

• Design, develop, and maintain robust ETL/ELT pipelines to ingest data from multiple sources (Oracle ERP, production systems, WMS, IoT sensors, external APIs).
• Build batch and real-time data pipelines using orchestration tools (Airflow, Prefect, or similar) to support analytics and AI/ML workloads.
• Ensure data quality, consistency, and reliability throughout the pipeline lifecycle.

• Data Warehousing & Lake Architecture for BI/AI:

• Design and manage data warehouse and data lake solutions (Snowflake, BigQuery, Redshift, or Azure Synapse) optimized for analytics and AI consumption.
• Implement dimensional modeling (Kimball/Inmon) and data vault methodologies to support BI dashboards and AI feature stores.
• Optimize storage, partitioning, and query performance for large-scale datasets used by data analysts and data scientists

• Data Preparation for AI/ML:

• Build and maintain feature stores and training datasets for machine learning models.
• Collaborate with data scientists to understand data requirements for predictive modeling, forecasting, and optimization algorithms.
• Implement data transformation logic for model training, validation, and inference pipelines.

• Data Integration & APIs:

• Integrate data from internal systems (Oracle ERP, WMS, HRIS) and external sources (supplier portals, logistics APIs, market data feeds).
• Build and maintain API-based data extraction and ingestion services for analytics consumption.
• Collaborate with application teams to ensure seamless data flow between operational systems and the analytics environment.

• Data Governance & Quality:

• Implement data quality checks, validation rules, and monitoring within pipelines to ensure accuracy for BI and AI use cases.
• Ensure data lineage, cataloging, and metadata management for full traceability.
• Collaborate with the Data Governance team to enforce data standards and security policies.

• Performance Optimization for Analytics:

• Tune SQL queries, pipeline performance, and data processing jobs to meet SLA requirements for dashboards and AI model refreshes.
• Optimize resource utilization and cost efficiency for cloud data platforms.
• Troubleshoot data pipeline failures and ensure timely resolution with minimal impact on analytics users.

• Collaboration & Support:

• Work closely with Data Analysts, BI Developers, and Data Scientists to understand data requirements for dashboards, reports, and models.
• Provide clean, well-documented, and timely datasets for analytics and AI/ML initiatives.
• Mentor junior data engineers and contribute to best practices within the Data & Analytics team.

• Documentation & Compliance:

• Maintain technical documentation for data pipelines, schemas, data dictionaries, and feature stores.
• Ensure compliance with data security, privacy, and retention policies.

Academic and Professional Qualifications :

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or related field.
• Master's degree is a plus.

Experience

• 5–8 years of experience in data engineering, ETL/ELT development, or data warehousing.
• Experience in manufacturing, FMCG, or food processing industries is preferred.
• Proven experience supporting analytics, BI, and AI/ML initiatives is highly desirable.
• Experience with ERP data extraction is a strong advantage.

Skills
• Technical Skills:

• Programming: Advanced Python (Pandas, PySpark) and SQL.
• ETL/Orchestration: Airflow, dbt, Prefect, or similar tools.
• Cloud Platforms: AWS (S3, Redshift, Glue), Azure (Data Factory, Synapse), or GCP (BigQuery, Dataflow).
• Data Warehousing: Snowflake, Big Query, Redshift, or Azure Synapse.
• Big Data Technologies: Spark, Kafka (preferred).
• Version Control: Git (GitHub/GitLab). o Containerization: Docker (preferred).
• Feature Stores: Experience with Feast, Tecton, or similar (preferred).

• Soft Skills:

• Strong analytical and problem-solving abilities.
• Excellent communication and collaboration skills, especially with analytics and data science teams.
• Attention to detail and commitment to data quality.
• Ability to work independently and handle multiple priorities.