Senior Data Engineer

3 semanas atrás

Brazil DeepRec.ai Tempo inteiro

Senior Data Engineer – ML Services and Data Pipelines

Location: Remote

Employment Type: Contract

Experience: 5+ years

3 month contract + multiple extensions possible


Our client is based in the USA but employs a remote workforce. You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone. Happy to hire remote worldwide.


We are seeking a skilled and experienced Data Engineer to design and implement scalable and extensible Machine Learning services and data pipelines within our AWS/Kubernetes environment. In this role, you will be responsible for setting up infrastructure to support the ingestion, processing, and indexing of text-based data. You will collaborate closely with our ML and software engineering teams to build robust pipelines and enable efficient data processing and document indexing.


Key Responsibilities


  • ML Infrastructure Setup: Design, implement, and maintain scalable ML infrastructure on AWS and Kubernetes to support current and future project needs. Tools like Amazon SageMaker for model development, deployment, and monitoring, and AWS Lambda for serverless computing will be key.
  • Data Pipeline Development: Develop, deploy, and manage data pipelines that process large volumes of data for machine learning use cases, with a focus on efficient text data processing. Use AWS Glue for ETL jobs, Amazon Kinesis for real-time data streaming, and AWS Step Functions to coordinate workflows.
  • Vector Database Integration: Set up and maintain vector databases to support natural language processing (NLP) models, ensuring efficient and accurate text-based data retrieval and analysis. Amazon OpenSearch Service and Amazon DynamoDB can be leveraged for indexing and storing large volumes of vectorized data.
  • Document Indexing System: Design and implement a system for document ingestion and indexing, providing seamless access to data for downstream ML and analytics processes. Tools like Amazon S3 for scalable storage and AWS Lambda for automation and processing will play a critical role.
  • Development Pipeline Setup: Establish a development pipeline for document ingestion, collaborating with DevOps and data science teams to ensure continuous integration and deployment practices using AWS CodePipeline and AWS CodeBuild.


Required Skills & Qualifications



  • Experience: 5+ years of experience in data engineering with a focus on ML infrastructure and data pipelines.
  • Technical Expertise:
  • Strong background in AWS and Kubernetes for deploying and managing scalable ML and data solutions.
  • Proficiency in data pipeline tools and frameworks such as AWS Glue, Amazon Kinesis, AWS Step Functions, or similar.
  • Experience with text-based data processing, NLP techniques, and vector databases (e.g., Amazon OpenSearch Service, Amazon DynamoDB, or third-party vector databases like Pinecone, Weaviate, or FAISS).
  • Programming Skills: Advanced skills in Python or similar languages for data engineering and ML pipeline development.
  • Data Handling & Storage: Proficiency in data storage solutions, including Amazon S3, AWS Redshift, Amazon RDS, and AWS Data Lake.
  • Problem-Solving Abilities: Ability to troubleshoot and resolve issues within ML pipelines and data processing environments.


Preferred Qualifications


  • Hands-on experience with ML Ops frameworks and practices.
  • Familiarity with continuous integration/continuous deployment (CI/CD) for data engineering workflows using AWS CodePipeline, AWS CodeBuild, and AWS CloudFormation.

  • Senior Data Engineer

    2 semanas atrás

    São Paulo, Brazil, BR BL Consultants Tempo inteiro

    Senior Data Engineer - São Paulo, Brazil - RemoteAbout the Role: Our client is looking for a Senior Data Engineer to be a technical powerhouse to help us scale our data infrastructure, automation and tools to meet growing business needs.You’re excited about this opportunity because you will…Work with business partners and stakeholders to understand data...

  • Senior Data Engineer

    Há 4 horas

    Brazil Zenode Tempo inteiro

    We’re an early stage startup looking to change the world!!! (shocking, right? 😂😇) We are building an AI copilot for electrical engineers to automate the monotonous, repetitive tasks in PCB design the same way that Github Copilot did for software engineers. We have trained a custom AI to read the component datasheets that serve as the instruction...

  • Data Engineer

    Há 7 dias

    Brazil, BR Insight Global Tempo inteiro

    Must-haves:5+ years of data engineer experience Experience working with AWS architecture (RDS, Glue, EMR, EC2, S3, Postgres, EMR, etc..)Snowflake data warehousingPython & SQL codingDay to Day:Insight Global is looking for 3 remote data engineers to join the analytics organization at a global medical device client. We are establishing a new data analytics...

  • Data Engineer

    Há 7 dias

    Brazil Insight Global Tempo inteiro

    Must-haves:5+ years of data engineer experience Experience working with AWS architecture (RDS, Glue, EMR, EC2, S3, Postgres, EMR, etc..)Snowflake data warehousingPython & SQL codingDay to Day:Insight Global is looking for 3 remote data engineers to join the analytics organization at a global medical device client. We are establishing a new data analytics...

  • Senior Data Engineer

    3 semanas atrás

    Brazil, BR DeepRec.ai Tempo inteiro

    Senior Data Engineer – ML Services and Data PipelinesLocation: RemoteEmployment Type: ContractExperience: 5+ years3 month contract + multiple extensions possibleOur client is based in the USA but employs a remote workforce. You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone. Happy to hire...

  • Data Engineer

    2 meses atrás

    Brazil Premier Group Recruitment Tempo inteiro

    Data EngineerContract$32 - $36 per hour. Premier Group has engaged exclusively with a growing marketing agency who are seeking a Data Engineer to join them on a 3-month contract, with a huge possibility of extension. The company is an English speaking business, so you must have strong English, both written and verbal, as well as your own Sociedade...

  • Data Engineer

    2 meses atrás

    Brazil Premier Group Recruitment Tempo inteiro

    Data Engineer Contract $32 - $36 per hour. Premier Group has engaged exclusively with a growing marketing agency who are seeking a Data Engineer to join them on a 3-month contract, with a huge possibility of extension. The company is an English speaking business, so you must have strong English, both written and verbal, as well as your own Sociedade...

  • Data Engineer

    2 meses atrás

    Brazil, BR Premier Group Recruitment Tempo inteiro

    Data EngineerContract$32 - $36 per hour. Premier Group has engaged exclusively with a growing marketing agency who are seeking a Data Engineer to join them on a 3-month contract, with a huge possibility of extension. The company is an English speaking business, so you must have strong English, both written and verbal, as well as your own Sociedade...

  • Data Engineer

    3 semanas atrás

    Brazil, BR Pivotal Solutions Tempo inteiro

    This is a contract position that will start immediately through the end of the year, with possible extension through next year depending on performance.Responsibilities:Collaborate with data analysts, engineers, business stakeholders, data scientists, and other team members to understand detailed data requirementsWrite code in Python and Airflow to build...

  • Data Engineer

    3 semanas atrás

    Brazil Pivotal Solutions Tempo inteiro

    This is a contract position that will start immediately through the end of the year, with possible extension through next year depending on performance.Responsibilities:Collaborate with data analysts, engineers, business stakeholders, data scientists, and other team members to understand detailed data requirementsWrite code in Python and Airflow to build...

  • Senior Data Engineer

    Há 1 mês

    Brazil HCLTech Tempo inteiro

    www.hcltech.comWe are HCLTech, one of the world’s largest and fastest growing technology and DSA companies with over 227,000 professionals across 60 countries, driving progress through industry-leading capabilities focused on Digital, Engineering and Cloud.The driving force behind this work, our people, is a diverse, creative and passionate audience that...

  • Data Engineer

    2 semanas atrás

    Brazil, BR Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Data Engineer

    2 semanas atrás

    Brazil Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Data Engineer

    2 semanas atrás

    Brazil Remobi Tempo inteiro

    Data Engineer - (REMOTE) Brazil About Remobi: We are building the world's greatest community of remote technologists! Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the...

  • Data Engineer

    3 meses atrás

    Brazil Remobi Tempo inteiro

    Data Engineer - (REMOTE) Brazil About Remobi: We are building the world's greatest community of remote technologists! Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the...

  • Data Engineer

    3 meses atrás

    Brazil Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Data Engineer

    3 meses atrás

    Brazil, BR Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Senior Python Engineer

    Há 1 mês

    Brazil, BR YASH Technologies Tempo inteiro

    Hello We are actively looking for a Senior Python Engineer. If you or your consultant are actively looking for a new job please share your profile.Role: Senior Python EngineerDuration: 2+ Months Location: Brazil ( REMOTE ) Mandatory Skills : Experience with Airflow on Kubernetes or KEDAWe are seeking a highly skilled Senior Python Engineer to join our...

  • Senior Python Engineer

    Há 1 mês

    Brazil YASH Technologies Tempo inteiro

    Hello We are actively looking for a Senior Python Engineer. If you or your consultant are actively looking for a new job please share your profile.Role: Senior Python EngineerDuration: 2+ Months Location: Brazil ( REMOTE ) Mandatory Skills : Experience with Airflow on Kubernetes or KEDAWe are seeking a highly skilled Senior Python Engineer to join our...

  • Senior Data Engineer

    Há 7 dias

    Brazil Blankfactor Tempo inteiro

    This is a remote position as a contractor paying in USD. Please apply with your English CV and only if you hold a minimum B1 English comprehension level.About BlankfactorAt Blankfactor, we are dedicated to engineering impact. Our passion lies in delivering best-in-class tech solutions that enable companies to transform, innovate, and scale. We specialize in...