August 04, 2020   |   3min read

Société Générale and Their Journey to Open Source

Do you remember my last blog post about big data for your business? No? Don’t worry, I won’t get offended (but just in case, you can find it on our website). This time, I will tell you a beautiful tale about an open-source project and two brave men I met around two weeks ago. I’m going to put a spotlight on two talented individuals and their path to Apache Airflow at Société Générale.

We’re right after the Airflow Summit that my colleagues from Polidea co-organized. It was on-line this time! You can still check out the archived talks from the Airflow Summit.

When I initially saw the list of the speakers, I knew I wanted to talk to at least a few of them off the record if I can say so. And because I know the right people (connections are everything ;) Thanks Jarek & Tomek!) I had a chance to meet Alaeddine Maaoui and Mohammed Marragh from Société Générale. They were happy to give me a sneak peek of their Airflow Summit talk and tell a story of their path to Apache Airflow. Trust me, I was far from being disappointed. I was actually amazed and, bear in mind, I do not get excited that easily. What Alaeddine and Mohammed told us then and during their talk at the Summit were valuable lessons useful to developers, managers, and business people. I noticed a tremendous change in their people’s mindset and the rapid growth of the teams working with Apache Airflow. The results of their strategy were exceptional, and 2020 is not finished yet.

What was the challenge at Société Générale?

First of all, SG (I’ll use this abbreviation from now on to describe Société Générale) had a huge legacy orchestration solution that wasn’t perfect for the financial market’s growing needs. The number of managed infrastructures and applications they showed us was way beyond my imagination.

The solution was not efficient, and they told themselves, “enough is enough, it’s time for a change.” Therefore, our colleagues from SG had to decide what to do next with this legacy solution, so that it’s reliable, interoperable, and, most importantly, secure. Does it already ring a bell?

Open-source tool Apache Airflow logo

Why Airflow?

At the beginning of 2020, they decided to migrate the infrastructure and applications to open source. Among other orchestration tools such as Luigi, Azkaban, Digdag—Apache Airflow was a natural choice for SG. Why, you may ask? It had all of the things they required:

  • Diagram scheduling
  • High availability
  • Native API
  • UI
  • Log management
  • Python
  • CI/CD
  • Resiliency
  • Scalability
  • Support
  • and it was free as a beer!

Choosing the right tool and implementing it was not the only bullet point in the SG’s strategy. In fact, they wanted to achieve something bigger. Their main intention was to change people’s mindset and make them contribute to open source. The open-source community is a great choice if you want your solutions to grow fast, be customizable as well as cooperate with kind and talented engineers. Such people are to be found in the Apache Airflow community. Also, when you pick an open-source solution with no SLA or dedicated support, you have to build an internal team that can solve the problems on their own as needed or be able to interact with companies providing support for such open-source solutions.

Changing the mindset

I won’t lie to you. Changing someone else’s mindset, especially an experienced developer’s one, is quite a challenge. I was amazed while listening to Mohammed and Alaeddine’s presentation. The main thing that struck me is that they didn’t want to force SG teams to switch to Airflow “just because”. They wanted to show them the benefits that the change may bring and let them know the tool better. That’s why they prepared three main areas that they presented to SG’s specialists team.

  1. Onboarding or else presenting Apache Airflow in a way that everyone knows it can be useful. They did it by carrying out live demos, workshops, or providing dev docker stack.
  2. Internal community (excellent idea btw). Developers at SG had a chance to try the tool themselves and create their own operators.
  3. Managed instances of Airflow.

However, was it enough to successfully carry on the journey to converting to open source at SG? You’ll find the answer to this question while watching Mohammed’s and Alaeddine’s talk at the Airflow Summit’s archive.

Karolina Rosół

Head of Cloud & Open Source

Did you enjoy the read?

If you have any questions, don’t hesitate to ask!