Senior AI Data Engineer
AI/ML
Remote
Contract
Title – Senior AI Data Engineer
Start date: Immediate
Position Type: Contract
Location: Remote across Canada/USA
About the Role:
We are looking for an experienced Senior AI Microservices and Data Engineer to design and implement scalable AI microservices and robust data pipelines. This role requires a strong background in data engineering, expertise in building microservices architecture for AI/ML models, and deep understanding of CI/CD pipelines integrated with MLOps workflows.
As a critical member of the team, you will ensure seamless integration between data engineering pipelines, AI services, and operational systems while supporting the continuous improvement and deployment of AI/ML models.
Key Responsibilities:
AI as Microservices:
- Design and develop AI/ML models as microservices using frameworks such as FastAPI, Flask, or Spring Boot.
- Implement APIs for AI models to enable real-time and batch inferencing in scalable environments.
- Optimize AI microservices for low latency, high availability, and scalability.
Data Engineering:
- Build and maintain robust data pipelines to support AI/ML workflows, ensuring data availability, consistency, and security.
- Integrate structured and unstructured data from various sources into data lakes or warehouses.
- Develop ETL/ELT processes and automate data transformation workflows using tools like Apache Airflow, Databricks, or Spark.
CI/CD Pipelines and MLOps Integration:
- Develop CI/CD pipelines to automate model training, testing, deployment, and monitoring using tools like GitLab Actions, Jenkins, or Azure DevOps.
- Implement MLOps practices to manage the lifecycle of AI/ML models, including versioning, monitoring, and retraining workflows.
- Ensure seamless integration of data pipelines with MLOps systems for continuous learning and model improvement.
Infrastructure Management:
- Collaborate with DevOps teams to deploy AI microservices in containerized environments using Docker and Kubernetes.
- Optimize deployment strategies for on-premise, cloud, or hybrid environments.
Monitoring and Optimization:
- Set up monitoring systems for data pipelines, AI microservices, and model performance using tools like Prometheus, Grafana, or Datadog.
- Continuously improve system performance, reliability, and scalability.
Collaboration:
- Work closely with data scientists, MLOps engineers, and DevOps teams to ensure data and AI systems are seamlessly integrated.
- Provide technical guidance and mentorship to junior engineers.
Key Requirements:
Experience:
- 7+ years of experience in data engineering or software development.
- At least 3 years of hands-on experience in deploying AI/ML models as microservices.
- Proven experience in building scalable data pipelines and integrating them with MLOps workflows.
Technical Skills:
Microservices Development:
- Strong expertise in building RESTful and gRPC APIs for AI/ML applications.
- Proficiency with frameworks like FastAPI, Flask, or Spring Boot.
Data Engineering:
- Expertise in tools like Apache Spark, Airflow, Databricks, or similar.
- Strong knowledge of data formats and integration (e.g., JSON, Parquet, Avro).
CI/CD and MLOps:
- Experience with CI/CD tools like GitLab Actions, Jenkins, or Azure DevOps.
- Knowledge of MLOps frameworks such as MLflow, Kubeflow, or Seldon Core.
Programming and Infrastructure:
- Proficiency in Python, Java, or Scala.
- Hands-on experience with containerization and orchestration tools like Docker and Kubernetes.
- Familiarity with cloud platforms (e.g., AWS, Azure, or GCP).
Preferred Skills:
- Experience in real-time data processing with tools like Kafka or Flink.
- Knowledge of model optimization techniques for deployment (e.g., TensorRT, ONNX).
- Familiarity with security and compliance practices for AI and data systems.
Soft Skills:
- Strong analytical and problem-solving abilities.
- Excellent communication and collaboration skills.
