Why customizing open-source cloud software?
1
Lower costs of maintenance achieved by migrating your existing systems to the cloud
2
Greater flexibility by connecting the customized solution to your systems and adapting it to your needs
3
Higher productivity of your development team thanks to one code and optimized efficiency of the systems
4
Enhanced security—with additional security levels you can ensure the success of your business
5
Competitive advantages—create unique solution with quality that stands out in the market
6
Engagement with open source communities—share your custom features and knowledge with other businesses
What can we help you with?
1
Review of your existing cloud tools and consultancy on the possible developments
2
Fast and efficient implementation of needed extensions
3
Cooperation with open source communities on improving the tools
Our official committers are contributing to
"Working on an Open Source project such as Apache Airflow is very demanding but also equally rewarding when you realize how many businesses use it every day and how fantastic people you interact with are."
Jarek Potiuk
Principal Software Engineer,
Polidea
Our projects
Open Source
Apache Airflow
An All-in-one Scheduler For Seamless Workflows
Being part of the open source community, Polidea developed and implemented an extensive set of operators for the Apache Airflow system to work with various cloud service providers. The team—including 2 Airflow committers— became contributors to the Open Source Airflow Project and provided 70+ operators for the Airflow DAGs, meeting the highest standards of open source projects.
Open Source
Apache Beam
A Testing Framework for the Open Source Community
Apache Beam is an open source project that enables running jobs on different data processing engines (runners) and developing code in multiple languages (SDKs). Our developers successfully implemented the testing framework, which helps users make more informed decisions regarding whether or not Beam is the right choice for their project. The framework allows writing integration and performance tests that use different SDKs, real data processing engine instances, data sources (databases, filesystems), and significant loads of data.
Société Générale and Their Journey to Open Source
Do you remember my last blog post about big data for your business? No? Don’t worry, I won’t get offended (but just in case, you can find it on our website) This time, I will tell you a beautiful tale about an open-source project and two brave men I met around two weeks ago. I’m going to put a spotlight on two talented individuals and their path to Apache Airflow at Société Générale.
We’re right after the Airflow Summit that my colleagues from Polidea co-organized. It was on-line this time! You can still check out the archived talks from the Airflow Summit.
When I initially saw the list of the speakers, I knew I wanted to talk to at least a few of them off the record if I can say so. And because I know the right people (connections are everything ;) Thanks Jarek & Tomek!) I had a chance to meet Alaeddine Maaoui and Mohammed Marragh from Société Générale. They were happy to give me a sneak peek of their Airflow Summit talk and tell a story of their path to Apache Airflow. Trust me, I was far from being disappointed. I was actually amazed and, bear in mind, I do not get excited that easily. What Alaeddine and Mohammed told us then and during their talk at the Summit were valuable lessons useful to developers, managers, and business people. I noticed a tremendous change in their people’s mindset and the rapid growth of the teams working with Apache Airflow. The results of their strategy were exceptional, and 2020 is not finished yet.
What was the challenge at Société Générale?
First of all, SG (I’ll use this abbreviation from now on to describe Société Générale) had a huge legacy orchestration solution that wasn’t perfect for the financial market’s growing needs. The number of managed infrastructures and applications they showed us was way beyond my imagination.
The solution was not efficient, and they told themselves, “enough is enough, it’s time for a change.” Therefore, our colleagues from SG had to decide what to do next with this legacy solution, so that it’s reliable, interoperable, and, most importantly, secure. Does it already ring a bell?
Why Airflow?
At the beginning of 2020, they decided to migrate the infrastructure and applications to open source. Among other orchestration tools such as Luigi, Azkaban, Digdag—Apache Airflow was a natural choice for SG. Why, you may ask? It had all of the things they required:
- Diagram scheduling
- High availability
- Native API
- UI
- Log management
- Python
- CI/CD
- Resiliency
- Scalability
- Support
- and it was free as a beer!
Choosing the right tool and implementing it was not the only bullet point in the SG’s strategy. In fact, they wanted to achieve something bigger. Their main intention was to change people’s mindset and make them contribute to open source. The open-source community is a great choice if you want your solutions to grow fast, be customizable as well as cooperate with kind and talented engineers. Such people are to be found in the Apache Airflow community. Also, when you pick an open-source solution with no SLA or dedicated support, you have to build an internal team that can solve the problems on their own as needed or be able to interact with companies providing support for such open-source solutions.
Changing the mindset
I won’t lie to you. Changing someone else’s mindset, especially an experienced developer’s one, is quite a challenge. I was amazed while listening to Mohammed and Alaeddine’s presentation. The main thing that struck me is that they didn’t want to force SG teams to switch to Airflow “just because”. They wanted to show them the benefits that the change may bring and let them know the tool better. That’s why they prepared three main areas that they presented to SG’s specialists team.
- Onboarding or else presenting Apache Airflow in a way that everyone knows it can be useful. They did it by carrying out live demos, workshops, or providing dev docker stack.
- Internal Community (excellent idea btw). Developers at SG had a chance to try the tool themselves and create their own operators.
- Managed instances of Airflow.
However, was it enough to successfully carry on the journey to converting to open source at SG? You’ll find the answer to this question while watching Mohammed’s and Alaeddine’s talk at the Airflow Summit’s archive.
