Spring Boot Kafka Producer Consumer Configuration, easy in 15 minutes

Today, the Spring Boot Kafka Producer Consumer Configuration tutorial walks you through the way that sends and receives messages from Spring Kafka. Here, you will configure Spring Kafka Producer and Consumer manually to know how Spring Kafka works. Let’s dig deeper.

Spring Boot Kafka Producer Consumer Configuration

Download and Install Apache Kafka

If you work on Windows platform, please read the post to know the way to download and install Apache Kafka. If you work on other platforms please read the official documentation here. So far, Apache Kafka is started on your local machine.

Project Setup

  • Spring Kafka: 2.3.4.RELEASE
  • Spring Boot: 2.2.2.RELEASE
  • Apache Kafka: kafka_2.13-2.4.0
  • Maven: 3.5

Project Structure

Spring Boot Kafka Producer Consumer Configuration

Maven Dependencies

At least, we must have these dependencies: spring-boot-starter, spring-kafka in pom.xml file. Note, you can change spring-kafka version for your own needs.

Configure Spring Kafka Application with application.yml

Some properties that need to be explicitly set in the application.yml file, e.g host, port, topic name:

Sending Kafka Messages to Topic

Spring Kafka introduces the KafkaTemplate class which wraps a Producer and provides high-level operations to send data to Kafka topics.

Another step that is KafkaTemplate configuration, KafkaTemplate is created from DefaultKafkaProducerFactory implementation. Some properties are required for that implementation:

  • ProducerConfig.BOOTSTRAP_SERVERS_CONFIG specifies a list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
  • ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG specifies the serializer class for key
  • ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG specifies the serializer class for value

You maybe want to refer to the list of configuration options take a look at the ProducerConfig class.

Here is the Spring Boot Kafka Producer configuration.

Receiving Kafka Messages from Topic

Next, we’ll implement Receiver class with receive() method that will consume messages of a Kafka topic. The receive() method is annotated with the @KafkaListener annotation.

Now we do the configuration for @KafkaListener with some properties:

  • ConsumerConfig.GROUP_ID_CONFIG specifies the consumer group, here is employee value.
  • ConsumerConfig.AUTO_OFFSET_RESET_CONFIG specifies which offset you wan to use:
    • earliest: automatically reset the offset to the earliest offset
    • latest: automatically reset the offset to the latest offset
    • none: throw exception to the consumer if no previous offset is found for the consumer’s group
    • anything else: throw exception to the consumer.
  • ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG set value as same as the SenderConfig class.
  • ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIGset value as same as the SenderConfig class.
  • ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIGset value as same as the SenderConfig class.

Here is the Spring Boot Kafka Consumer configuration:

Spring Boot Application

Now we have done Spring Boot Kafka Producer Consumer configuration, then we create a Spring Boot application to demonstrate the application like below.

Running the Application

Before we make a demo for the Spring Boot Apache Kafka example, Kafka server must start on localhost with port 9092 which is the default configuration of Kafka. Then we run Spring boot application normally and see the output.

That’s all about Spring Boot Kafka Producer Consumer Configuration. You should download the complete source code and run on your local machine to get more deeply the way Kafka works. You can also create Spring Boot Kafka Producer and Consumer without configuration, let check out the post Spring Boot Apache Kafka Example

Download complete source code

spring-kafka-producer-consumer-example.zip (31 downloads)

References

Apache Kafka Official Website
Spring Kafka Client Compatability
Spring Kafka Documentation
Spring Boot Tutorial for Beginners

Please share it if you found this useful
Hide Buttons