How to Install Apache Kafka on Ubuntu 18.04

r00t July 14, 2018

Install Apache Kafka on Ubuntu 18.04

In this tutorial we are going to learn how to install Apache Kafka on Ubuntu 18.04. We will also install and configure its prerequisites. Apache Kafka is an open-source scalable and high-throughput messaging system developed by the Apache Software Foundation written in Scala. Apache Kafka is specially designed to allow a single cluster to serve as the central data backbone for a large environment. It has a much higher throughput compared to other message brokers’ systems like ActiveMQ and RabbitMQ. It is capable of handling large volumes of real-time data efficiency. You can deploy Kafka on single Apache server or in a distributed clustered environment.

I recommend to use a minimal Ubuntu server setup as a basis for the tutorial, that can be a virtual or a root server image with an Ubuntu 18.04 Bionic Beaver minimal install from a web hosting company or you use our minimal server tutorial to install a server from scratch.

Install Apache Kafka on Ubuntu 18.04

Step 1. First, ensure your system and apt package lists are fully up-to-date by running the following:

apt-get update -y
apt-get upgrade -y

Step 2. Install Java.

Kafka is written in Java and Scala and requires jre 1.7 and above to run it. In this step, we will ensure Java is installed:

apt-get install openjdk-8-jdk

You can verify that JDK 8 is installed properly by running the following command:

sudo java -version

Step 3. Install Zookeeper.

Before installing Apache Kafka, you will need to have zookeeper available and running. ZooKeeper is an open source service for maintaining configuration information, providing distributed synchronization, naming and providing group services:

sudo apt-get install zookeeperd

Once installation is finished, it will be started as a daemon automatically, we will now check if Zookeeper is alive and if it’s OK.

netstat -ant | grep :2181

Step 4. Installing Apache Kafka on Ubuntu 18.04.

Run the following command to download Apache Kafka:

wget http://www-eu.apache.org/dist/kafka/1.1.0/kafka_2.12-1.1.0.tgz

Optionally check the integrity of the downloaded file:

curl http://kafka.apache.org/KEYS | gpg --import
wget https://dist.apache.org/repos/dist/release/kafka/1.1.0/kafka_2.12-1.1.0.tgz.asc
gpg --verify kafka_2.12-1.1.0.tgz.asc kafka_2.12-1.1.0.tgz

Next, Create a directory for extracting Kafka:

sudo mkdir /opt/kafka
sudo tar -xvzf kafka_2.12-1.1.0.tgz --directory /opt/kafka --strip-components 1

Step 5. Configuring Apache Kafka Server.

Kafka persists data to disk so we will now make a directory for it:

sudo mkdir /var/lib/kafka
sudo mkdir /var/lib/kafka/data

Set permission to all of the Kafka related directories:

sudo chown -R kafka:nogroup /opt/kafka
sudo chown -R kafka:nogroup /var/lib/kafka

Then, open /opt/kafka/config/server.properties in nano editor:

sudo nano /opt/kafka/config/server.properties

Find the line and change it:

delete.topic.enable = true
log.dirs=/var/lib/kafka/data
log.retention.hours=168 #other accepted keys are(log.retention.ms, log.retention.minutes)

Step 6. Testing Kafka Server.

Now, it is time to verify the Kafka server is operating correctly:

sudo /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

In another terminal create a topic:

/opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

List all topics with the command below and it will print test the topic we just created:

/opt/kafka/bin/kafka-topics.sh --list --zookeeper localhost:2181

Let’s start publishing messages on test topic:

/opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

We will now create a subscriber on test topic and listen from the beginning of the topic:

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Enter some message in the producer:

Hello world!!!

Step 7. Configure Kafka as a service on startup.

Create a unit file in /etc/systemd/system directory:

sudo nano /etc/systemd/system/kafka.service

Adds following  lines:

[Unit]
Description=High-available, distributed message broker
After=network.target
[Service]
User=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
[Install]
WantedBy=multi-user.target

Start the newly created service:

sudo systemctl start kafka.service
sudo systemctl enable kafka.service

Congratulation’s! You have successfully install and configured Apache Kafka on your Ubuntu 18.04 LTS server. Thanks for using this tutorial installing Apache Kafka on Ubuntu 18.04 LTS (Bionic Beaver) system.

The Tags:

Leave a Comment

Comments are closed.