{"componentChunkName":"component---src-pages-markdown-remark-fields-slug-js","path":"/engineering/quick-kafka-installation/","result":{"data":{"markdownRemark":{"id":"1f3f868c-85de-556a-b0e6-50a8006931c6","excerpt":"In this post, we will look at the step-by-step process for Kafka Installation on Windows. Kafka is an open-source stream-processing software platform and comes…","html":"

In this post, we will look at the step-by-step process for Kafka Installation on Windows. Kafka is an open-source stream-processing software platform and comes under the Apache software foundation.

\n

What is Kafka?

\n

Kafka is used for real-time streams of data, to collect big data, or to do real-time analysis (or both). Kafka is used with in-memory microservices to provide durability and it can be used to feed events to complex event streaming systems and IoT/IFTTT-style automation systems.

\n

Installation :

\n

1. Java Setup:

\n

Kafka requires Java 8 for running. And hence, this is the first step that we should do to install Kafka. To install Java, there are a couple of options. We can go for the Oracle JDK version 8 from the Official Oracle Website.

\n

2. Kafka & Zookeeper Configuration:

\n

Step 1: Download Apache Kafka from its Official Site.

\n

Step 2: Extract tgz via cmd or from the available tool to a location of your choice:

\n
tar -xvzf kafka_2.12-2.4.1.tgz
\n

Step 3: Copy the path of the Kafka folder. Now go to config inside Kafka folder and open zookeeper.properties file. Copy the path against the field dataDir and add /zookeeper-data to the path.

\n

\nStep 4: we have to modify the config/server.properties file. Below is the change:

\n
fileslog.dirs=C:\\kafka\\kafka-logs
\n

Basically, we are pointing the log.dirs to the new folder /data/kafka.

\n

Run Kafka Server:

\n

Step 1: Kafka requires Zookeeper to run. Basically, Kafka uses Zookeeper to manage the entire cluster and various brokers. Therefore, a running instance of Zookeeper is a prerequisite to Kafka.

\n

To start Zookeeper, we can open a PowerShell prompt and execute the below command:

\n
.\\bin\\windows\\zookeeper-server-start.bat .\\config\\zookeeper.properties
\n

If the command is successful, Zookeeper will start on port 2181.

\n

Step 2: Now open another command prompt and change the directory to the kafka folder. Run kafka server using the command:

\n
.\\bin\\windows\\kafka-server-start.bat .\\config\\server.properties
\n

Now your Kafka Server is up and running, you can create topics to store messages. Also, we can produce or consume data directly from the command prompt.

\n

Create a Kafka Topic:

\n
    \n
  1. Open a new command prompt in the location C:\\kafka\\bin\\windows.
    2. \n
  2. Run the following command:
    3. \n
\n
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
\n

Creating Kafka Producer:

\n
    \n
  1. Open a new command prompt in the location C:\\kafka\\bin\\windows
    2. \n
  2. Run the following command:
    3. \n
\n
kafka-console-producer.bat --broker-list localhost:9092 --topic test
\n

Creating Kafka Consumer:

\n
    \n
  1. Open a new command prompt in the location C:\\kafka\\bin\\windows.
    2. \n
  2. Run the following command:
    3. \n
\n
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning
\n

If you see these messages on consumer console,Congratulations!!! you all done. Then you can play with producer and consumer terminal bypassing some Kafka messages.

\n","headings":[{"value":"What is Kafka?","depth":2},{"value":"Installation :","depth":2},{"value":"1. Java Setup:","depth":3},{"value":"2. Kafka & Zookeeper Configuration:","depth":3},{"value":"Run Kafka Server:","depth":2},{"value":"Create a Kafka Topic:","depth":2},{"value":"Creating Kafka Producer:","depth":2},{"value":"Creating Kafka Consumer:","depth":2}],"fields":{"slug":"/engineering/quick-kafka-installation/"},"frontmatter":{"metatitle":null,"metadescription":null,"description":null,"title":"Setting Up and Running Apache Kafka on Windows OS","canonical":null,"date":"August 25, 2020","updated_date":null,"tags":["Kafka","Windows"],"coverImage":{"childImageSharp":{"fluid":{"aspectRatio":1.5037593984962405,"src":"/static/41484554ead7eb70ade1abe9498282b0/2ad7f/messagelog.webp","srcSet":"/static/41484554ead7eb70ade1abe9498282b0/1c9b5/messagelog.webp 200w,\n/static/41484554ead7eb70ade1abe9498282b0/f1752/messagelog.webp 400w,\n/static/41484554ead7eb70ade1abe9498282b0/2ad7f/messagelog.webp 800w,\n/static/41484554ead7eb70ade1abe9498282b0/e7405/messagelog.webp 1200w,\n/static/41484554ead7eb70ade1abe9498282b0/d3cba/messagelog.webp 1600w,\n/static/41484554ead7eb70ade1abe9498282b0/772d3/messagelog.webp 2700w","sizes":"(max-width: 800px) 100vw, 800px"}}},"author":{"id":"Ashish Sharma","github":"ashish8947","bio":"Ashish is a software engineer who continually seeks clean, elegant solutions to business challenges. His lifelong passion for technology drives him to proactively expand horizons, constantly exploring and evaluating new languages, platforms, frameworks and tools.","avatar":null}}}},"pageContext":{"id":"1f3f868c-85de-556a-b0e6-50a8006931c6","fields__slug":"/engineering/quick-kafka-installation/","__params":{"fields__slug":"engineering"}}},"staticQueryHashes":["1171199041","1384082988","1711371485","1753898100","2100481360","229320306","23180105","528864852"]}