Setting up Kafka

Before deploying Kafka, you should have followed the steps in Preparing for Kafka.

If you’re deploying to a real cluster with many physical machines, it’s best to deploy with three Kafka replicas to get the feel for a realistic production setup. The spec below deploys three Kafka replicas, one ZooKeeper, and uses ephemeral storage (so if your pods are destroyed, they lose their data). The Shopping Cart example contains this configuration in the file deploy/kafka.yaml.

apiVersion: kafka.strimzi.io/v1alpha1
kind: Kafka
metadata:
  name: strimzi
spec:
  kafka:
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

If you’re deploying to Minishift, you’ll likely find that by the time all the ZooKeeper and Kafka instances are deployed, along with all the auxiliary services deployed by the operator, your machine has no resources left for anything else, such as your application, to use. So instead you can use a spec that only deploys one Kafka replica and one ZooKeeper replica. In addition, we’ll need to change the replication factors to one, since with only one replica, we can’t replicate more than once. The following shows the relevant sections of the deploy\kafka-single.yaml

...
  kafka:
    replicas: 1

...
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
...

Deploying with three replicas

Enter the following command to deploy with the kafka.yaml file:

oc apply -f deploy/kafka.yaml -n $NAMESPACE

Deploying to Minishift with one replica

Enter the following command to deploy with the kafka-single.yaml file:

oc apply -f deploy/kafka-single.yaml -n $NAMESPACE

Viewing the deployment

Once you’ve deployed your Kafka instance, you can watch it come up by running:

oc get pods -w -n myproject

You should eventually see something like the following output, with the number of Kafka and ZooKeeper pods corresponding to the number of replicas you configured.

strimzi-entity-operator-6bc7f6985c-q29p5   3/3     Running   0          44s
strimzi-kafka-0                            2/2     Running   1          91s
strimzi-kafka-1                            2/2     Running   1          91s
strimzi-kafka-2                            2/2     Running   1          91s
strimzi-zookeeper-0                        2/2     Running   0          2m30s
strimzi-cluster-operator-78f8bf857-kpmhb   1/1     Running   0          3m10s

It’s also useful to see what services have been deployed:

oc get services -n myproject

This should show at least:

strimzi-kafka-bootstrap    ClusterIP   172.30.231.126   <none>        9091/TCP,9092/TCP,9093/TCP,9404/TCP   2m
strimzi-kafka-brokers      ClusterIP   None             <none>        9091/TCP,9092/TCP,9093/TCP            2m
strimzi-zookeeper-client   ClusterIP   172.30.226.168   <none>        9404/TCP,2181/TCP                     3m
strimzi-zookeeper-nodes    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP            3m

As you can see, there is a service called strimzi-kafka-brokers, this is the service that Kafka clients are going to connect to.

Once Kafka is deployed and running, you no longer need to be logged in as an administrator, so log back in as your old user. If using Minishift, that means logging in as the developer user:

oc login -u developer

What’s next

With the database and Kafka running, the next step is Configuring Shopping Cart.