With this ETL platform, users can extract their data collected by third-party analytical sources like Adwords, Facebook, Google Analytics, Google Management, or Big Query, transform it (visualize) as they see fit, and upload it wherever they want. This service offers users to pick the data to be shown and extracted, how often it should be extracted, and how it should look in the end.
The platform was lacking an admin panel, was written with AWS but in a way that stalled the scaling, and the database couldn’t process & transform users’ data fast enough. Plus, some of the platform’s logic was written into the database. We had to:
Re-do the architecture
Speed up data processing
Develop admin panel
Untangle the platform logic from the database into the backend
We’ve replaced the Monolith that used to rely on one Lambda to perform all functions with Microservices architecture that relies on multiple Lambda to perform them independently, ensuring that the platform performs as promised without failures in one Lambda inflicting failures onto the others.
We’ve transferred the project from JS Framework to Python, which made data processing faster and opened an opportunity to increase its quality by adding supportive AI algorithms. And in the process, we’ve picked apart the work of the service logic from Postgres DB and migrated it to the backend with Python.
We’ve designed and developed a web app with an admin panel where users can easily add their sources, set up the periods of data extraction, select how the data should get transformed, see server-usage statistics.
Further developed the platform’s database so that the user’s analytical context would be uploaded and transformed according to selected pre-settings.
Increased the speed of the data processing
Improved the quality of code that can be easily scalable and supportable
Made platform secure so users could trust it with their data
APIs, website, and security enhancement
Architecture Improvements
AWS is the next best thing after sliced bread
Lambda Team © 2017 - 2023
This site is owned and operated by Aleksandr Blokh
+38 073 00 89 171 | [email protected]