What does it take to create a real-time dashboard app to prepare and display aggregated streaming data? This article explores a sample application that leverages powerful frameworks, cloud services, and infrastructure to achieve this with little effort.

Application

Aggregate and display top Twitter hashtags within a specific time window in real-time.

In a nutshell, the application receives data from Twitter Filtered Stream API, pushes to Kafka topics, and aggregates with Dataflow Beam app to be consumed by Streamlit powered UI.

High-Level Architecture

In this application, we are using Twitter Filtered Stream API as the data source. It is easy to set up and…


Apache Beam and Google Dataflow are helping data engineers adapt Kappa architecture with a unified programming model and a managed service with a production scale runner. In enterprise solutions, data processing jobs are part of complex pipelines, and fully productionizing these jobs means integration with existing Continuous Integration(CI) and automation for Continuous Deployment(CD). This article explores an event-based Dataflow job automation approach using Cloud Composer, Airflow, and Cloud Functions.

Motivation

Google Cloud Platform(GCP) documentation provides reference solutions for setting up a CI/CD pipeline and scheduling Dataflow jobs. However, these solutions do not provide a simple interface and abstraction from the underlying…


This article explains a DAG (Directed Acyclic Graph) runner implementation using Scala Futures and Promises. The runner can execute actions in parallel and is not limited by dependency depth.

DAG has many applications, from data pipelines to CD pipelines. DAGs are how dependencies can be represented, ensuring no cyclic references. These are common requirements at every software stack, and in fact, there are countless frameworks and implementation of DAG runners from a single server to distributed executions. For example, a build system needs to look at the project dependencies and define an order, i.e., topological sorting. …

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store