Create unique cloud solution thanks to open source customization!


Extend your open source cloud software and adjust it to you specific needs to boost productivity of your team and lower costs of maintenance. Learn how else open source customization can benefit your business!

Why customizing open-source cloud software?

1

2

3

4

5

6

Let’s talk about your project!

Need help with customizing open-source cloud software, like Apache Beam or Airflow? Our experienced developers, GCP Architects and Apache Committers are here for you!

What can we help you with?

1

2

3

Our official committers are contributing to

"Working on an Open Source project such as Apache Airflow is very demanding but also equally rewarding when you realize how many businesses use it every day and how fantastic people you interact with are."

Jarek Potiuk

Principal Software Engineer,
Polidea

Our projects

Open Source

Apache Airflow


An All-in-one Scheduler For Seamless Workflows

Being part of the open source community, Polidea developed and implemented an extensive set of operators for the Apache Airflow system to work with various cloud service providers. The team—including 2 Airflow committers— became contributors to the Open Source Airflow Project and provided 70+ operators for the Airflow DAGs, meeting the highest standards of open source projects.

Open Source

Apache Beam


A Testing Framework for the Open Source Community

Apache Beam is an open source project that enables running jobs on different data processing engines (runners) and developing code in multiple languages (SDKs). Our developers successfully implemented the testing framework, which helps users make more informed decisions regarding whether or not Beam is the right choice for their project. The framework allows writing integration and performance tests that use different SDKs, real data processing engine instances, data sources (databases, filesystems), and significant loads of data.

Our clients
are our partners

Société Générale and Their Journey to Open Source

Do you remember my last blog post about big data for your business? No? Don’t worry, I won’t get offended (but just in case, you can find it on our website) This time, I will tell you a beautiful tale about an open-source project and two brave men I met around two weeks ago. I’m going to put a spotlight on two talented individuals and their path to Apache Airflow at Société Générale.

We’re right after the Airflow Summit that my colleagues from Polidea co-organized. It was on-line this time! You can still check out the archived talks from the Airflow Summit.

When I initially saw the list of the speakers, I knew I wanted to talk to at least a few of them off the record if I can say so. And because I know the right people (connections are everything ;) Thanks Jarek & Tomek!) I had a chance to meet Alaeddine Maaoui and Mohammed Marragh from Société Générale. They were happy to give me a sneak peek of their Airflow Summit talk and tell a story of their path to Apache Airflow. Trust me, I was far from being disappointed. I was actually amazed and, bear in mind, I do not get excited that easily. What Alaeddine and Mohammed told us then and during their talk at the Summit were valuable lessons useful to developers, managers, and business people. I noticed a tremendous change in their people’s mindset and the rapid growth of the teams working with Apache Airflow. The results of their strategy were exceptional, and 2020 is not finished yet.

What was the challenge at Société Générale?

First of all, SG (I’ll use this abbreviation from now on to describe Société Générale) had a huge legacy orchestration solution that wasn’t perfect for the financial market’s growing needs. The number of managed infrastructures and applications they showed us was way beyond my imagination.

The solution was not efficient, and they told themselves, “enough is enough, it’s time for a change.” Therefore, our colleagues from SG had to decide what to do next with this legacy solution, so that it’s reliable, interoperable, and, most importantly, secure. Does it already ring a bell?

Why Airflow?

At the beginning of 2020, they decided to migrate the infrastructure and applications to open source. Among other orchestration tools such as Luigi, Azkaban, Digdag—Apache Airflow was a natural choice for SG. Why, you may ask? It had all of the things they required:

  • Diagram scheduling
  • High availability
  • Native API
  • UI
  • Log management
  • Python
  • CI/CD
  • Resiliency
  • Scalability
  • Support
  • and it was free as a beer!

Choosing the right tool and implementing it was not the only bullet point in the SG’s strategy. In fact, they wanted to achieve something bigger. Their main intention was to change people’s mindset and make them contribute to open source. The open-source community is a great choice if you want your solutions to grow fast, be customizable as well as cooperate with kind and talented engineers. Such people are to be found in the Apache Airflow community. Also, when you pick an open-source solution with no SLA or dedicated support, you have to build an internal team that can solve the problems on their own as needed or be able to interact with companies providing support for such open-source solutions.

Changing the mindset

I won’t lie to you. Changing someone else’s mindset, especially an experienced developer’s one, is quite a challenge. I was amazed while listening to Mohammed and Alaeddine’s presentation. The main thing that struck me is that they didn’t want to force SG teams to switch to Airflow “just because”. They wanted to show them the benefits that the change may bring and let them know the tool better. That’s why they prepared three main areas that they presented to SG’s specialists team.

  1. Onboarding or else presenting Apache Airflow in a way that everyone knows it can be useful. They did it by carrying out live demos, workshops, or providing dev docker stack.
  2. Internal Community (excellent idea btw). Developers at SG had a chance to try the tool themselves and create their own operators.
  3. Managed instances of Airflow.

However, was it enough to successfully carry on the journey to converting to open source at SG? You’ll find the answer to this question while watching Mohammed’s and Alaeddine’s talk at the Airflow Summit’s archive.