It's been almost 1 year since I've adopted Apache Airflow to my company. This post would be a quick lookback for this decision. Although the Airflow is in action, it has been only used as an alternative to the cron daemon. One of the biggest reasons was that every team members were busy to handling… Continue reading A year with the Apache Airflow
Is January 1st, 2023 (Sunday) the 1st week of 2023 or the last week of 2022? At first glance, this looks like a very simple question but actually it is not. Because "a week is a period of 7 days" is the only definition we have about a week. Does a week start on Sunday?… Continue reading ISO-8601 Week Definition: Is January 1st, 2023 the Last Week of 2022?
While working with databases, we might be in a situation to solve the problems related with handling max values or min values. "What product has been sold at highest price in last 5 years?", "Since 2000, who has hit the most home-runs in a season and which year was it?" or "On which date each… Continue reading Solving Max and Min Value Problems with SQL Window Functions
Although Airflow can send an email upon task failure, sometime it's not good enough for you. You may have other kind of communication channels like Slack, or you may just want to post it to the SNS like Twitter. In this case, on_success_callback and on_failure_callback could be a good choice. These callback functions are called… Continue reading Enhance Airflow Task Monitoring with Custom Callback Functions and Webhooks
Apache Airflow is a platform to programmatically author, schedule and monitor workflows. For whom has lots of scheduled jobs to run, Airflow is quite a mind-blowing platform. Although it also has some weird points, I think the good overcomes the bad. Among the many characteristics of Airflow, let's focus on job scheduling here. Actually, setting… Continue reading Understanding Airflow Job Scheduling: Timetable Concept Explained
"Hello, World" is the message being used when you are learning a software language for the first time. Almost every software language provides a small program printing this message1) to illustrate its basic syntax. Although I have almost 20 years of experiences in software development and data engineering, most of my experiences are from non-English… Continue reading Hello, World
Kirin's Technology Essay 