Straight to prod: How I test my streams

Testing stream processing applications (Kafka Streams and ksqlDB) isn’t always straightforward. You could run a simple topology manually and observe the results. How about repeatable tests that you can run anytime, as part of a build without a Kafka cluster or Zookeeper? Luckily, Kafka Streams includes the TopologyTestDriver module (and ksqlDB includes test-runner) that allows you to do precisely that. After learning this, no doubt, your test coverage is sky-high! However, how will your stream processing application perform once deployed to production? You might depend on external resources such as databases, web services, and connectors.

Viktor will start this talk covering the basics of unit testing of Kafka Streams applications using TopologyTestDriver. Viktor will also look at some popular open source libraries for testing streams applications. Viktor demonstrates TestContainers, a Java library that provides lightweight, disposable instances of shared databases, Kafka clusters, and anything else that can run in a Docker container and how to use it for integration testing of processing applications! And lastly, Viktor will show ksqlDB’s test-runner to unit test your KSQL applications.

Viktor Gamov is a Developer Advocate at Confluent, the company that makes a streaming platform based on Apache Kafka.

Working in the field, Viktor Gamov developed comprehensive expertise in building enterprise application architectures using open source technologies. He enjoys helping different organizations design and develop low latency, scalable and highly available distributed systems.

Back in his consultancy days, he co-authored O'Reilly's "Enterprise Web Development".

He is a professional conference speaker on distributed systems, Java, and JavaScript topics, and is regular on events including JavaOne, Devoxx, OSCON, QCon, and others (http://lanyrd.com/gamussa). He blogs at http://gamov.io and produces the podcasts Razbor Poletov (in Russian) and co-hosts DevRelRad.io.

Follow Viktor on Twitter @gamussa, where he posts about gym life, food, open source, and, of course, Kafka and Confluent!

Victor Gamov Confluent