Data Engineer Sr. Associate

Há 3 dias

São Paulo, Brasil Credit Saison Brazil Tempo integral
The Senior Data & AI Engineer is the hands-on owner of CSBR's lending data lake on AWS and a full-stack builder of the data products around it. Reporting to the Data Lead, the Engineer runs the pipelines and lakehouse that feed portfolio, market, and credit-decision data; uses AI coding agents to ship the applications that put that data to work; and contributes to credit modelling and data-science research, working closely with the investment, credit, finance, and product teams and with platform engineering. Data Lake & Pipelines
• Own the lending data lake end to end – a lakehouse of open-format tables on S3, Glue, and Athena – and the orchestrated pipelines on ECS Fargate that load it, from raw ingestion to analytical Gold tables and serving snapshots.
• Build and maintain ingestion from public and market sources (BCB, B3, CVM, IBGE, MTE, Receita Federal, PGFN) and from internal systems, partners, and fund administrators, including scraped and semi-structured data.
• Build the pipelines that bring data from CSBR's proprietary lending products into the lake, with stricter handling of personal data (PII): minimization, masking or pseudonymization, segregated access, and audit trails.
• Model the datasets behind FIDC portfolio valuation and covenant monitoring, market intelligence, vehicle pricing and risk, and credit-decision features, each with a table contract, lineage, and documentation. Data Quality & Operations
• Guarantee quality and freshness with data contracts, automated quality checks, reconciliation against source systems, and alerting, and lead the fix when a pipeline or a number is wrong.
• Deliver every change through automated tests, infrastructure as code, and CI/CD, with gated deploys and auditable evidence, and keep Athena, Fargate, and storage costs under control. Data Products & AI-Assisted Development
• Build and evolve the applications on top of the lake – the SQL query portal and its interface for AI assistants, the FIDC portfolio monitoring app, and the market-intelligence and vehicle dashboards – as web front ends and APIs running serverless on Lambda and Amplify.
• Develop LLM features on Amazon Bedrock – the portal's data assistant, covenant extraction from fund documents – with evaluation sets, cost tracking, and release gates. Data Science & Modelling
• Prepare model-ready data – point-in-time features, labels, and reference populations – and run batch scoring and model monitoring as pipelines in the lake.
• Partner with the Data Lead on research and development of credit and employer-risk models: exploratory analysis, feature engineering, fitting, validation, and calibration, with tracked, reproducible experiments under the team's modelling protocol.
• Contribute to CSBR's long-horizon data-science research project, turning research prototypes into reproducible, production-grade code. Collaboration & Special Projects
• Turn questions from the investment, credit, and finance teams into datasets, analyses, and tools, and explain the results clearly to non-technical colleagues.
• Set engineering standards for data work – pipeline guides, templates, code review – and mentor the colleagues who contribute to the lake.
• Support new lending products as they launch, and other initiatives as prioritized by the Data Lead. Requirements
• Native or bilingual Portuguese and professional English, equivalent to CEFR B2 or above.
• Degree in Computer Science, Engineering, Statistics, Mathematics, or a related field, or equivalent experience.
• 5+ years in data engineering or closely related roles, including owning production pipelines on a cloud data lake or lakehouse; financial-services experience – banks, fintechs, lenders, or asset managers – preferred.
• Expert Python and SQL: tested, typed, well-packaged code, and analytical SQL over large, partitioned datasets in Athena or a comparable query engine.
• Production experience with workflow orchestration and lakehouse design: dependency-aware scheduling, partitioned and incremental loads, backfills, idempotent runs, open table formats, schema evolution, and layered Bronze/Silver/Gold modelling.
• Solid AWS for data workloads (S3, Glue, Athena, Lambda, ECS Fargate, DynamoDB, IAM), delivered through infrastructure as code and CI/CD.
• Ability to ship a web application end to end with AI assistance: API design, authentication, a component-based front end, and automated deployment.
• Working data-science foundation: statistics, supervised learning with linear and gradient-boosted models, validation and calibration, and reproducible experiment tracking.
• Understanding of credit and lending data – portfolios, cash flows, delinquency, provisioning – and of LGPD; familiarity with Brazilian public data (BCB, CVM, IBGE, Receita Federal) is a plus.
• Autonomy and clear communication: able to own a problem from question to production in a small team, document decisions, and explain data to business stakeholders.
• Desirable: credit scoring or credit-risk modelling; FIDC or securitization analytics; LLM applications on Amazon Bedrock, tool-using agents, and LLM evaluation; entity resolution or graph analysis on company data (CNPJ); SageMaker; AWS data or machine-learning certifications.