org.apache.flume.sink.kafka

Class KafkaSink

java.lang.Object

org.apache.flume.sink.AbstractSink

org.apache.flume.sink.kafka.KafkaSink

All Implemented Interfaces:

Configurable, LifecycleAware, NamedComponent, Sink

public class KafkaSink
extends AbstractSink
implements Configurable

A Flume Sink that can publish messages to Kafka. This is a general implementation that can be used with any Flume agent and a channel. The message can be any event and the key is a string that we read from the header For use of partitioning, use an interceptor to generate a header with the partition key

Mandatory properties are: brokerList -- can be a partial list, but at least 2 are recommended for HA

however, any property starting with "kafka." will be passed along to the Kafka producer Read the Kafka producer documentation to see which configurations can be used

Optional properties topic - there's a default, and also - this can be in the event header if you need to support events with different topics batchSize - how many messages to process in one batch. Larger batches improve throughput while adding latency. requiredAcks -- 0 (unsafe), 1 (accepted by at least one broker, default), -1 (accepted by all brokers) useFlumeEventFormat - preserves event headers when serializing onto Kafka

header properties (per event): topic key

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flume.Sink

Sink.Status

Constructor Summary

Constructors

Constructor and Description
KafkaSink()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(Context context) We configure the sink and generate properties for the Kafka Producer Kafka producer properties is generated as follows: 1.
int	getBatchSize()
protected Properties	getKafkaProps()
String	getTopic()
Sink.Status	process() Requests the sink to attempt to consume data from attached channel
void	start() Starts a service or component.
void	stop() Stops a service or component.

Methods inherited from class org.apache.flume.sink.AbstractSink

getChannel, getLifecycleState, getName, setChannel, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

KafkaSink

public KafkaSink()

Method Detail

getTopic

public String getTopic()

getBatchSize

public int getBatchSize()

process

public Sink.Status process()
                    throws EventDeliveryException

Description copied from interface: Sink

Requests the sink to attempt to consume data from attached channel

Note: This method should be consuming from the channel within the bounds of a Transaction. On successful delivery, the transaction should be committed, and on failure it should be rolled back.

Specified by:

process in interface Sink

Returns:

READY if 1 or more Events were successfully delivered, BACKOFF if no data could be retrieved from the channel feeding this sink

Throws:

EventDeliveryException - In case of any kind of failure to deliver data to the next hop destination.

start

public void start()

Description copied from interface: LifecycleAware

Starts a service or component.

Implementations should determine the result of any start logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

start in interface LifecycleAware

Overrides:

start in class AbstractSink

stop

public void stop()

Description copied from interface: LifecycleAware

Stops a service or component.

Implementations should determine the result of any stop logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

stop in interface LifecycleAware

Overrides:

stop in class AbstractSink

configure

public void configure(Context context)

We configure the sink and generate properties for the Kafka Producer Kafka producer properties is generated as follows: 1. We generate a properties object with some static defaults that can be overridden by Sink configuration 2. We add the configuration users added for Kafka (parameters starting with .kafka. and must be valid Kafka Producer properties 3. We add the sink's documented parameters which can override other properties

Specified by:

configure in interface Configurable

Parameters:

context -

getKafkaProps

protected Properties getKafkaProps()