Context

• Alma is a French fintech leader in the Buy Now Pay Later industry
• To drive its complex operations Alma needed truthful and reliable dashboards, an integration of third parties tools and efficients tools for their Business Analyst and Data Scientist.
• Before our collaboration Alma was building dashboards on top of their applicative database and raw data. This was hard to maintain and time consuming for their analysts. Data Scientists were lacking proper tool and know-how to streamline their work and build top notch risk assessment algorithms. Data ingestion and transformation was done at the same time.
• In over 8 months we built a Modern Data Stack from scratch, integrated all their data sources, built layered BI transformations and delivered a high quality framework for business analysts and data scientists. We also built real time ingestion pipeline of their applicative data, ensuring it was always fresh.
• The end-result was much more efficient reports, automated tedious tasks and streamlined workflows for virtually every team. The fraud assessment algorithm built on top of the Data Platform allowed to avoid around 50M€ yearly of fraudulent transactions.

Data Engineering

Data Engineering is the first step to build a Modern Data Stack, this layer is responsible for

• Ingestion of external data (Datalake)
• Exporting data onto third parties tools (Reverse ETL)
• Data Privacy
• Maintenance of the data infrastructure

Datalake

I integrated a variety of raw data sources into their Datalake

• Postgres (applicative database)
• Hubspot
• Zendesk
• Stripe
• Adyen
• Google Sheets

I ensured that ingestion was fault resilient and truthful by providing dashboards that compared data quality between the Analytics database and the source of truth.

Their Postgres database was first ingested using a custom batch workflow but as they needed real-time data we ultimately switched to real time ingestion using CDC (Capture Data Change).

I monitored their infrastructure every day, ensuring quick fixes when any upstream ingestion workflow was broken.

Export data to third parties

An important need for Alma was to be able to deliver transformed data into external tools used by business teams. I created custom Reverse connectors to deliver data to Hubspot, Zendesk, or Customer.io built upon their respective APIs.

I created dedicated environments for third parties, so that they could deliver data directly onto their Analytics Database without compromising production data.

Data Privacy

Alma is a fintech that works with many businesses and direct clients, they pay businesses directly on behalf of clients so they can then pay Alma in installments. A data leak could therefore compromise many actors at the same and it was from upmost importance to ensure only people that needed specific datasets could access it.

I used GCP Big Query tag system to tag tables and columns according to their security level thus only people having specific acreditation could access sensitive data.

Infrastructure Maintenance

• Managing data stack using Terraform
• Ensuring third parties would have an access to our cloud
• Role of Site Reliability Engineer within the data team

Analytics Engineering

Analytics Engineering is the layer where we craft a robust framework for data manipulation, it is responsible for,

• Transformation of raw data into higher level business objects (modelization)
• Building informative dashboards for business owners
• Ensuring transformations are fault-tolerant

Transformation of raw data

Raw data is transformed into refined business objects that are tangible for analysts and stakeholders, this transformation is done within an hierarchy of layers. The business layer will contains table that describe business object in a holistic way.

We built this hierarchy for them and worked together with business analyst and business stakeholders to build relevant business layer.

We implemented those transformations using DBT, a powerful tool to transform your data. We then onboarded business analysts and data scientists on this infrastructure so they could be autonomous and build their own models without the help of data engineers.

This layered architecture makes it 10x easier for business analysts to build dashboards. After building it we migrated existing dashboard onto this new architecture.

Fault tolerant transformations

We implemented defensive programming with statistics check that would run in the CI github workflow so that if an updated column would have a standard deviation shift, or any statistics that would be a fit then the Pull Request could not be merged automatically.

The CI would also kickstart a Docker compose with Airflow and DBT and if a failure were to arise the merging would be blocked. This ensure safe and streamlined developpement processes.

Data Science

The main focus of this project was Data and Analytics engineering but I also contributed to Data Science efforts by supporting the team on a daily basis. I was responsible for coaching them on Data Engineering and Software Engineering practices.

We brainstormed and industrialized together their risk assessment model that would decide whether a transaction should be accepted or not.

An A/B test revealed that the new version of that model was responsible for a savings up to 50M€ a year, this was the culminant achievement of Alma's Data Platform and it would have never been possible without a powerful and robust Data Engineering infrastructure.