Lower costs of maintenance achieved by migrating your existing systems to the cloud
Greater flexibility by connecting the customized solution to your systems and adapting it to your needs
Higher productivity of your development team thanks to one code and optimized efficiency of the systems
Enhanced security—with additional security levels you can ensure the success of your business
Competitive advantages—create unique solution with quality that stands out in the market
Engagement with open source communities—share your custom features and knowledge with other businesses
Review of your existing cloud tools and consultancy on the possible developments
Fast and efficient implementation of needed extensions
Cooperation with open source communities on improving the tools
"Working on an Open Source project such as Apache Airflow is very demanding but also equally rewarding when you realize how many businesses use it every day and how fantastic people you interact with are."
Principal Software Engineer,
Being part of the open source community, Polidea developed and implemented an extensive set of operators for the Apache Airflow system to work with various cloud service providers. The team—including 2 Airflow committers— became contributors to the Open Source Airflow Project and provided 70+ operators for the Airflow DAGs, meeting the highest standards of open source projects.
Apache Beam is an open source project that enables running jobs on different data processing engines (runners) and developing code in multiple languages (SDKs). Our developers successfully implemented the testing framework, which helps users make more informed decisions regarding whether or not Beam is the right choice for their project. The framework allows writing integration and performance tests that use different SDKs, real data processing engine instances, data sources (databases, filesystems), and significant loads of data.
Do you remember my last blog post about big data for your business? No? Don’t worry, I won’t get offended (but just in case, you can find it on our website) This time, I will tell you a beautiful tale about an open-source project and two brave men I met around two weeks ago. I’m going to put a spotlight on two talented individuals and their path to Apache Airflow at Société Générale.
We’re right after the Airflow Summit that my colleagues from Polidea co-organized. It was on-line this time! You can still check out the archived talks from the Airflow Summit.
When I initially saw the list of the speakers, I knew I wanted to talk to at least a few of them off the record if I can say so. And because I know the right people (connections are everything ;) Thanks Jarek & Tomek!) I had a chance to meet Alaeddine Maaoui and Mohammed Marragh from Société Générale. They were happy to give me a sneak peek of their Airflow Summit talk and tell a story of their path to Apache Airflow. Trust me, I was far from being disappointed. I was actually amazed and, bear in mind, I do not get excited that easily. What Alaeddine and Mohammed told us then and during their talk at the Summit were valuable lessons useful to developers, managers, and business people. I noticed a tremendous change in their people’s mindset and the rapid growth of the teams working with Apache Airflow. The results of their strategy were exceptional, and 2020 is not finished yet.
First of all, SG (I’ll use this abbreviation from now on to describe Société Générale) had a huge legacy orchestration solution that wasn’t perfect for the financial market’s growing needs. The number of managed infrastructures and applications they showed us was way beyond my imagination.
The solution was not efficient, and they told themselves, “enough is enough, it’s time for a change.” Therefore, our colleagues from SG had to decide what to do next with this legacy solution, so that it’s reliable, interoperable, and, most importantly, secure. Does it already ring a bell?
At the beginning of 2020, they decided to migrate the infrastructure and applications to open source. Among other orchestration tools such as Luigi, Azkaban, Digdag—Apache Airflow was a natural choice for SG. Why, you may ask? It had all of the things they required:
Choosing the right tool and implementing it was not the only bullet point in the SG’s strategy. In fact, they wanted to achieve something bigger. Their main intention was to change people’s mindset and make them contribute to open source. The open-source community is a great choice if you want your solutions to grow fast, be customizable as well as cooperate with kind and talented engineers. Such people are to be found in the Apache Airflow community. Also, when you pick an open-source solution with no SLA or dedicated support, you have to build an internal team that can solve the problems on their own as needed or be able to interact with companies providing support for such open-source solutions.
I won’t lie to you. Changing someone else’s mindset, especially an experienced developer’s one, is quite a challenge. I was amazed while listening to Mohammed and Alaeddine’s presentation. The main thing that struck me is that they didn’t want to force SG teams to switch to Airflow “just because”. They wanted to show them the benefits that the change may bring and let them know the tool better. That’s why they prepared three main areas that they presented to SG’s specialists team.
However, was it enough to successfully carry on the journey to converting to open source at SG? You’ll find the answer to this question while watching Mohammed’s and Alaeddine’s talk at the Airflow Summit’s archive.