Senior Data Engineer

3 semanas atrás

brasília, Brasil DeepRec.ai Tempo inteiro
Senior Data Engineer – ML Services and Data Pipelines
Location:
RemoteEmployment Type:
ContractExperience:
5+ years3 month contract + multiple extensions possible
Our client is based in the USA but employs a remote workforce. You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone. Happy to hire remote worldwide.
We are seeking a skilled and experienced
Data Engineer
to design and implement scalable and extensible Machine Learning services and data pipelines within our AWS/Kubernetes environment. In this role, you will be responsible for setting up infrastructure to support the ingestion, processing, and indexing of text-based data. You will collaborate closely with our ML and software engineering teams to build robust pipelines and enable efficient data processing and document indexing.
Key Responsibilities
ML Infrastructure Setup
: Design, implement, and maintain scalable ML infrastructure on AWS and Kubernetes to support current and future project needs. Tools like
Amazon SageMaker
for model development, deployment, and monitoring, and
AWS Lambda
for serverless computing will be key.Data Pipeline Development
: Develop, deploy, and manage data pipelines that process large volumes of data for machine learning use cases, with a focus on efficient text data processing. Use
AWS Glue
for ETL jobs,
Amazon Kinesis
for real-time data streaming, and
AWS Step Functions
to coordinate workflows.Vector Database Integration
: Set up and maintain vector databases to support natural language processing (NLP) models, ensuring efficient and accurate text-based data retrieval and analysis.
Amazon OpenSearch Service
and
Amazon DynamoDB
can be leveraged for indexing and storing large volumes of vectorized data.Document Indexing System
: Design and implement a system for document ingestion and indexing, providing seamless access to data for downstream ML and analytics processes. Tools like
Amazon S3
for scalable storage and
AWS Lambda
for automation and processing will play a critical role.Development Pipeline Setup
: Establish a development pipeline for document ingestion, collaborating with DevOps and data science teams to ensure continuous integration and deployment practices using
AWS CodePipeline
and
AWS CodeBuild
.
Required Skills & Qualifications
Experience
: 5+ years of experience in data engineering with a focus on ML infrastructure and data pipelines.Technical Expertise
:Strong background in AWS and Kubernetes for deploying and managing scalable ML and data solutions.Proficiency in data pipeline tools and frameworks such as
AWS Glue
,
Amazon Kinesis
,
AWS Step Functions
, or similar.Experience with text-based data processing, NLP techniques, and vector databases (e.g.,
Amazon OpenSearch Service
,
Amazon DynamoDB
, or third-party vector databases like Pinecone, Weaviate, or FAISS).Programming Skills
: Advanced skills in Python or similar languages for data engineering and ML pipeline development.Data Handling & Storage
: Proficiency in data storage solutions, including
Amazon S3
,
AWS Redshift
,
Amazon RDS
, and
AWS Data Lake
.Problem-Solving Abilities
: Ability to troubleshoot and resolve issues within ML pipelines and data processing environments.
Preferred Qualifications
Hands-on experience with ML Ops frameworks and practices.Familiarity with continuous integration/continuous deployment (CI/CD) for data engineering workflows using
AWS CodePipeline
,
AWS CodeBuild
, and
AWS CloudFormation
.

  • Data Engineer

    Há 7 dias

    Brasília, Brasil Insight Global Tempo inteiro

    Must-haves:5+ years of data engineer experienceExperience working with AWS architecture (RDS, Glue, EMR, EC2, S3, Postgres, EMR, etc..)Snowflake data warehousingPython & SQL codingDay to Day:Insight Global is looking for 3 remote data engineers to join the analytics organization at a global medical device client. We are establishing a new data analytics...

  • Data Engineer

    Há 7 dias

    Brasília, Brasil Insight Global Tempo inteiro

    Must-haves: 5+ years of data engineer experienceExperience working with AWS architecture (RDS, Glue, EMR, EC2, S3, Postgres, EMR, etc..)Snowflake data warehousingPython & SQL codingDay to Day:Insight Global is looking for 3 remote data engineers to join the analytics organization at a global medical device client. We are establishing a new data analytics...

  • Senior Data Engineer(Scala)

    2 meses atrás

    Brasília, Brasil Sigma Software Tempo inteiro

    Company Description Are you passionate about data and eager to make a significant impact in the ad tech industry? We're seeking a Senior Data Engineer to be a key player in our dynamic team. Our focus is on transforming and optimizing high-velocity ad log streams, creating a unified data lake for seamless downstream data consumption. Our client processes...

  • Senior Data Engineer

    1 semana atrás

    Brasília, Brasil DataArt Solutions, Inc Tempo inteiro

    Remote. ColombiaRemote. LATAMClient Our client is a huge investment company headquartered in New York City.Join a great company, not merely an individual project.Project Overview The primary objective of this exciting project is to enhance the functionality of a cutting-edge Data Platform, empowering business users with the insights they need to make...

  • Senior Data Engineer

    3 semanas atrás

    Brasília, Brasil DeepRec.ai Tempo inteiro

    Senior Data Engineer – ML Services and Data PipelinesLocation: RemoteEmployment Type: ContractExperience: 5+ years3 month contract + multiple extensions possibleOur client is based in the USA but employs a remote workforce. You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone. Happy to hire...

  • Senior Data Engineer

    3 semanas atrás

    Brasília, Brasil DeepRec.ai Tempo inteiro

    Senior Data Engineer – ML Services and Data PipelinesLocation:RemoteEmployment Type:ContractExperience:5+ years3 month contract + multiple extensions possibleOur client is based in the USA but employs a remote workforce. You should be available for team meetings at 8am PT and have some cross over for team collaboration in PST/ET timezone. Happy to hire...

  • Data Engineer

    3 semanas atrás

    Brasília, Brasil Pivotal Solutions Tempo inteiro

    This is a contract position that will start immediately through the end of the year, with possible extension through next year depending on performance.Responsibilities:Collaborate with data analysts, engineers, business stakeholders, data scientists, and other team members to understand detailed data requirementsWrite code in Python and Airflow to build...

  • Data Engineer

    2 meses atrás

    brasília, Brasil Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Data Engineer

    1 semana atrás

    brasília, Brasil Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Data Engineer

    3 meses atrás

    Brasília, Brasil Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Data Engineer

    2 meses atrás

    Brasília, Brasil Premier Group Recruitment Tempo inteiro

    Data EngineerContract$32 - $36 per hour.Premier Group has engaged exclusively with a growing marketing agency who are seeking a Data Engineer to join them on a 3-month contract, with a huge possibility of extension.The company is an English speaking business, so you must have strong English, both written and verbal, as well as your own Sociedade...

  • Data Engineer

    4 semanas atrás

    Brasília, Distrito Federal, Brasil Blueclip Tempo inteiro

    About BlueclipWe are a team of creative minds and experts who deliver great works. Our journey started with a vision to bring together the most talented individuals and cutting-edge technology to drive innovation.Job DescriptionWe are seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will play a crucial role in facilitating the...

  • Data Engineer

    2 semanas atrás

    Brasília, Brasil Remobi Tempo inteiro

    Data Engineer - (REMOTE) BrazilAbout Remobi:We are building the world's greatest community of remote technologists!Today, organizations that understand the value of remote working will reap the rewards. It doesn’t just provide team members with a healthier work-life balance, it gives you the opportunity to access the brightest minds in the world.Our...

  • Systems Analyst

    2 semanas atrás

    Brasília, Brasil ZF Friedrichshafen AG Tempo inteiro

    Are you ready to make an impact? ZF is looking for talented individuals to join our team. As a FutureStarter, you’ll have the opportunity to shape the future of mobility. Join us and be part of something extraordinary!Country/Region: BRLocation: Sorocaba, São Paulo, BR, 18103-905About the Data Engineer: As a Data Engineer, you are part of a small...

  • Senior Data Engineer

    Há 1 mês

    brasília, Brasil HCLTech Tempo inteiro

    We are HCLTech, one of the world’s largest and fastest growing technology and DSA companies with over 227,000 professionals across 60 countries, driving progress through industry-leading capabilities focused on Digital, Engineering and Cloud.The driving force behind this work, our people, is a diverse, creative and passionate audience that enables us to...

  • Senior Data Engineer

    Há 1 mês

    Brasília, Brasil HCLTech Tempo inteiro

    We are HCLTech, one of the world’s largest and fastest growing technology and DSA companies with over 227,000 professionals across 60 countries, driving progress through industry-leading capabilities focused on Digital, Engineering and Cloud.The driving force behind this work, our people, is a diverse, creative and passionate audience that enables us to...

  • Data Engineer

    3 semanas atrás

    Brasília, Brasil Pivotal Solutions Tempo inteiro

    This is a contract position that will start immediately through the end of the year, with possible extension through next year depending on performance.Responsibilities:Collaborate with data analysts, engineers, business stakeholders, data scientists, and other team members to understand detailed data requirementsWrite code in Python and Airflow to build...

  • Senior Software Engineer

    2 semanas atrás

    Brasília, Brasil Ignitho Technologies Inc. Tempo inteiro

    Position Details: Senior Software Engineer (.Net) Location: Brazil, BrazilOpenings: 1Salary Range:Description:Develop and maintain .NET applications hosted on Azure cloud services, with a focus on Azure Kubernetes Services.Design and optimize SQL databases, and proficiently execute complex queries for efficient data retrieval and manipulation.Configure...

  • Senior Devops Engineer

    3 semanas atrás

    Brasília, Brasil Vingcard Tempo inteiro

    TITLE:Senior DevOPS EngineerLOCATION/GEOGRAPHY:Brasília, BrazilReports To:R&D ManagerRole Summary:As a Senior DevOps Engineer, you will play a pivotal role in managing and optimizing cloud infrastructure, ensuring the seamless integration of software development tools, and maintaining the security and reliability of

  • Insight Global | Data Engineer | brasília

    Há 6 dias

    brasília, Brasil Insight Global Tempo inteiro

    Must-haves:5+ years of data engineer experienceExperience working with AWS architecture (RDS, Glue, EMR, EC2, S3, Postgres, EMR, etc..)Snowflake data warehousingPython & SQL codingDay to Day:Insight Global is looking for 3 remote data engineers to join the analytics organization at a global medical device client. We are establishing a new data analytics...