Data Engineer w/Gen AI
Jersey City, NJ
Contracted
Experienced
This is a Hybrid role with 3 days a week in the office
We are seeking an experienced Data Engineer with 3-5+ years of experience to join our dynamic team. The ideal candidate will possess a strong background in data management and engineering, specifically within cloud environments. You will play a crucial role in developing and managing data pipelines that support AI and Generative AI initiatives, ensuring that our data architecture is robust, scalable, and optimized for performance.
Key Responsibilities:
We are seeking an experienced Data Engineer with 3-5+ years of experience to join our dynamic team. The ideal candidate will possess a strong background in data management and engineering, specifically within cloud environments. You will play a crucial role in developing and managing data pipelines that support AI and Generative AI initiatives, ensuring that our data architecture is robust, scalable, and optimized for performance.
Key Responsibilities:
- Data Pipeline Development: Design, develop, and manage data pipelines to support AI and Generative AI data requirements.
- Workflow Creation: Build self-service onboarding workflows in data federation platforms, particularly using AWS Athena, to facilitate efficient data access and integration.
- Schema Management: Own the ingestion of schemas, metadata APIs (including table schema descriptions), and table registration services to enhance data governance.
- SQL Execution Layer Design: Design and implement a SQL execution layer via AWS Athena that optimizes query performance and ensures data integrity.
- Access Controls: Implement table access controls, audit logging, and schema diffing to maintain security and compliance across our data assets.
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and ensure alignment with organizational goals.
- Continuous Improvement: Identify opportunities for process enhancements and drive best practices in data engineering and management.
- Proficient in SQL and experience with AWS services, particularly Athena.
- Strong experience in ETL processes and data pipeline development.
- Proficiency in Python for data manipulation and automation tasks.
- Familiarity with REST APIs and Git for version control and collaboration.
- Understanding of IAM basics and data access control principles.
- 3-5+ years of experience in data engineering or a related field.
- Bachelor’s degree in Computer Science, Information Technology, or a related discipline (or equivalent experience).
- Strong analytical and problem-solving skills, with a keen attention to detail.
- Excellent communication skills and ability to work collaboratively in a team environment.
Apply for this position
Required*