Apache Beam: A Basic Guide
Apache Beam simplifies large-scale data processing dynamics. Let’s read more about the features, basic concepts, and the fundamentals of Apache beam.
We'll talk about Apache Beam in this guide and discuss its fundamental concepts. We will begin by showing the features and advantages of using Apache Beam, and then we will cover basic concepts and terminologies.
Ever since the concept of big data got introduced to the programming world, a lot of different technologies, frameworks have emerged. The processing of data can be categorized into two different paradigms. One is Batch Processing, and the other is Stream Processing.
Different technologies came into existence for different paradigms, solving various big data world problems, for, e.g., Apache Spark, Apache Flink, Apache Storm, etc.
As a developer or a business, it's always challenging to maintain different tech stacks and technologies. Hence, Apache Beam to the rescue!
What is Apache Beam?
Apache Beam is an open source, centralised model for describing parallel-processing pipelines for both batch and streaming data. The programming model of the Apache Beam simplifies large-scale data processing dynamics.
The Apache Beam model offers helpful abstractions that insulate you from distributed processing information at low levels, such as managing individual staff, exchanging databases, and other activities. These low-level information are handled entirely by Dataflow.
Features of Apache Beam
The unique features of Apache beam are as follows:
- Unified - Use a single programming model for both batch and streaming use cases.
- Portable - Execute pipelines in multiple execution environments. Here, execution environments mean different runners. Ex. Spark Runner, Dataflow Runner, etc
- Extensible - Write custom SDKs, IO connectors, and transformation libraries.
Apache Beam SDKs and Runners
As of today, there are 3 Apache beam programming SDKs
- Java
- Python
- Golang
Beam Runners translate the beam pipeline to the API compatible backend processing of your choice. Beam currently supports runners that work with the following backends.
- Apache Spark
- Apache Flink
- Apache Samza
- Google Cloud Dataflow
- Hazelcast Jet
- Twister2
Direct Runner to run on the host machine, which is used for testing purposes.
Basic Concepts in Apache Beam
Apache Beam has three main abstractions. They are
- Pipeline
- PCollection
- PTransform
Pipeline:
A pipeline is the first abstraction to be created. It holds the complete data processing job from start to finish, including reading data, manipulating data, and writing data to a sink. Every pipeline takes in options/parameters that indicate where and how to run.
PCollection:
A pcollection is an abstraction of distributed data. A pcollection can be bounded, i.e., finite data, or unbounded, i.e., infinite data. The initial pcollection is created by reading data from the source. From then on, pcollections are the source and sink of every step in the pipeline.
Transform:
A transform is a data processing operation. A transform is applied on one or more pcollections. Complex transforms have other transform nested within them. Every transform has a generic method where the logic of the transform sits in.
Example of Pipeline
Here, let's write a pipeline to output all the jsons where the name starts with a vowel.
Let's take a sample input. Name the file as
The input should be a newline delimited JSON.
Include the following dependencies in your
Let's code the beam pipeline. Follow the steps
-
Create a pipeline.
Create a pipeline which binds all the pcollections and transforms. Optionally you can pass the PipelineOptions if needed.
-
Read the input file
Use the transform to read the input files. Every line is a different json record.
-
Apply a transform to filter out the names starting from a vowel
The filter transform takes a SerializableFunction Object where the method is overridden. Every json-string record is converted to a JSON. The first character of the is checked if it's a vowel. The transform is applied to each input JSON record. Based on the boolean value returned, the record is retained or discarded.
-
Write the results to a file
The results of the transform are stored in a text file using the write method of the transform. As PCollections are distributed across machines, the results are written to multiple files/shards. To avoid this, we use where all the output is written to a single file.
Output:
Complete Code:
For more advanced concepts, refer to the official site - beam.apache.org
