Apache Flume Tutorial: Streaming Twitter Data

Now as we know how Apache Flume works, let us take a look at a practical where we will sink the Twitter data and store it in the HDFS. Sink, collects the data from the channel and commits or writes the data in the HDFS permanently. Similarly, channel acts as the local storage or a temporary storage between the source of data and persistent data in the HDFS. Basically, the buffer acts as a intermediary storage that stores the data being transferred temporarily and therefore prevents data loss. Thus, we need some buffer to match the read & write speed difference. Channel: In general, the reading speed is faster than the writing speed.Source: It accepts the data from the incoming streamline and stores the data in the channel.The flume agent has 3 components: source, sink and channel. Twitter is among one of the famous sources for streaming data. From the diagram, you can easily understand that the web server indicates the data source. There is a Flume agent which ingests the streaming data from various data sources to HDFS. Now, let us understand the architecture of Flume from the below diagram: The architecture is one which is empowering Apache Flume with these benefits. Now, as we know the advantages of Apache Flume, lets move ahead and understand Apache Flume architecture.Īpache Flume Tutorial: Flume Architecture It supports a large set of sources and destinations types.It helps us to ingest online streaming data from various sources like network traffic, social media, email messages, log files etc.It gives us a solution which is reliable and distributed and helps us in collecting, aggregating and moving large amount of data sets like Facebook, Twitter and e-commerce websites.

Using Flume, we can ingest data from multiple servers into Hadoop.

The transactions in Flume are channel-based where two transactions (one sender & one receiver) are maintained for each message.

Flume provides reliable message delivery.

If the read rate exceeds the write rate, Flume provides a steady flow of data between read and write operations.

Apache Flume can store data in centralized stores (i.e data is supplied from a single store) like HBase & HDFS.

Flume is scalable, reliable, fault tolerant and customizable for different sources and sinks.

There are several advantages of Apache Flume which makes it a better choice over others. Apache Flume Tutorial: Advantages of Apache Flume Then moving ahead, we will look at the architecture of Flume and try to understand how it works fundamentally. It is fault-tolerant and provides reliability mechanism for Fault tolerance & failure recovery.Īfter understanding what is Flume, now let us advance in this Flume Tutorial blog and understand the benefits of Apache Flume. It has simple and flexible architecture based on streaming data flows. The main idea behind the Flume’s design is to capture streaming data from various web servers to HDFS. Flume is a highly reliable & distributed. It collects, aggregates and transports large amount of streaming data such as log files, events from various sources like network traffic, social media, email messages etc. Apache Flume Tutorial: Introduction to Apache FlumeĪpache Flume is a tool for data ingestion in HDFS. Then moving ahead, we will understand the advantages of using Flume.

We will be beginning this Flume tutorial by discussing about what is Apache Flume. In this Apache Flume tutorial blog, we will be covering: This is why Apache Flume is an important part of Hadoop Ecosystem. Flume can easily integrate with Hadoop and dump unstructured as well as semi-structured data on HDFS, complimenting the power of Hadoop. I hope you may be familiar with Apache Hadoop, which is being used tremendously in the industry as it can store all kinds of data. Lets talk about another important reason why Flume became so popular. There are be multiple sources from which data is gathered in an organization. Data ingestion is the initial & important step in order to process & analyse data, and then derive business values out of it. But before that let us understand the importance of data ingestion. In this Apache Flume tutorial blog, we will understand how Flume helps in streaming data from various sources.

Apache flume