jackdaw.client
Clojure wrapper to Kafka's consumer and producer APIs.
The consumers and producers are the basis for streams, and many
other use cases. They can be used to send messages directly to, or
read messages from topics. There are also some facilities for
polling, and transactions.
See `jackdaw.client.*` for some add-ons atop this API.
assign
(assign consumer & topic-partitions)
Assign a consumer to specific partitions for specific topics. Returns the consumer.
assign-all
(assign-all consumer topics)
Assigns all of the partitions for all of the given topics to the consumer
assignment
(assignment consumer)
Return the assigned topics and partitions of a consumer.
callback
(callback on-completion)
Return a kafka `Callback` function out of a clojure `fn`.
The fn must be of 2-arity, being `[record-metadata?, ex?]` where the
record-metadata may be the datafied metadata for the produced
record, and the ex may be an exception encountered while producing
the record.
Callbacks are `void`, so the return value is ignored.
consumer
(consumer config)(consumer config {:keys [key-serde value-serde], :as t})
Return a consumer with the supplied properties and optional Serdes.
num-partitions
(num-partitions producer-or-consumer topic)
Given a producer or consumer and a topic, return the number of
partitions for that topic.
Note that partitions are 0-indexed, so a number of partitions 1
means that only partition 0 exists.
offsets-for-times
(offsets-for-times consumer partition-timestamps)
Given a subscribed consumer and a mapping of topic-partition or
`TopicPartition` records to timestamps, return a mapping from
topic-partition descriptors to the offset into each partition of the
FIRST record whose timestamp is equal to or greater than the given
timestamp.
Timestamps are longs to MS precision in UTC.
partitions-for
(partitions-for producer-or-consumer {:keys [topic-name]})
Given a producer or consumer and a Jackdaw topic descriptor, return
metadata about the partitions assigned to the given consumer or
producer.
poll
(poll consumer timeout)
Polls kafka for new messages, returning a potentially empty sequence
of datafied messages.
position
(position consumer topic-partition)
Get the offset of the next record that will be fetched.
Accepts either a `TopicPartition` record, or a datafied
`TopicPartition` as generated by the rest of the Jackdaw API.
position-all
(position-all consumer)
Call position on every assigned partition, producing a map from
partition information to the consumer's offset into that partition.
produce!
(produce! producer topic value)(produce! producer topic key value)(produce! producer topic partition key value)(produce! producer topic partition timestamp key value)(produce! producer topic partition timestamp key value headers)
Helper wrapping `#'send!`.
Builds and sends a `ProducerRecord` so you don't have to. Returns
a future which will produce datafied record metadata when forced.
producer
(producer config)(producer config {:keys [key-serde value-serde]})
Return a producer with the supplied properties and optional Serdes.
seek
(seek consumer topic-partition offset)
Seek the consumer to the specified offset on the specified partition.
Accepts either a `TopicPartition` instance or a datafied
`TopicPartition` as produced by the rest of the Jackdaw API.
Returns the consumer for convenience with `->`, `doto` etc.
seek-to-beginning-eager
(seek-to-beginning-eager consumer)(seek-to-beginning-eager consumer topic-partitions)
Seek to the first offset for the given topic/partitions and force positioning.
When no partitions are passed, seek on all assigned
topic-partitions.
seek-to-end-eager
(seek-to-end-eager consumer)(seek-to-end-eager consumer topic-partitions)
Seek to the last offset for all assigned partitions, and force positioning.
When no partitions are passed, seek on all assigned partitions.
Returns the consumer.
seek-to-timestamp
(seek-to-timestamp consumer timestamp topics)
Given an timestamp in epoch MS, a subscribed consumer and a seq of
Jackdaw topics, seek all partitions of the selected topics to the
offsets reported by Kafka to correlate with the given timestamp.
After seeking, the first message read from each partition will be
the EARLIEST message whose timestamp is greater than or equal to the
timestamp sought.
Returns the consumer for convenience with `->`, `doto` etc.
send!
(send! producer record)(send! producer record callback-fn)
Asynchronously sends a record to a topic, returning a `Future`
which will produce a data structure describing the metadata of the
produced record when forced.
A 2-arity callback function may be provided. It will be invoked with
either [RecordMetdata, nil] or [nil, Exception] respectively if the
record was sent or if an exception was encountered.
subscribe
(subscribe consumer topic-configs)
Subscribe a consumer to the specified topics.
Returns the consumer.
subscribed-consumer
(subscribed-consumer config topic-configs)
Given a broker configuration and topics, returns a consumer that is
subscribed to all of the given topic descriptors.
WARNING: All topics subscribed to by a single consumer must share a
single pair of key and value serde instances. The serdes of the
first requested topic are used, and all other topics are expected to
be able to use same serdes.
subscription
(subscription consumer)
Return the subscription(s) of a consumer as a collection of topics.
Subscriptions are a set of strings, being the names of topics which
are subscribed to.