Send data from Apache Flume to ElasticSearch -


i using following flume 1.7 agent configuration stream data kafka 0.9.0.1 topic, , send data elasticsearch setup on rancher using es found in catalog version v0.5.0.

agent.sources  = flume_test agent.channels = kafka_test_channel agent.sinks =   elasticsearchsink  agent.sources.flume_test.type = org.apache.flume.source.kafka.kafkasource agent.sources.flume_test.zookeeperconnect = stage-kafka01.stage:2181,stage-kafka02.stage:2181,stage-kafka03.stage:2181 agent.sources.flume_test.topic = hrzn_stage1_test agent.sources.flume_test.groupid = flume_kafka_stage1_test agent.sources.flume_test.channels = kafka_test_channel agent.sources.flume_test.spooldir = /var/log/spooldir/auth agent.sources.flume_test.interceptors = i1 agent.sources.flume_test.interceptors.i1.type = com.cpm.interceptors.testinterceptor$builder  agent.channels.kafka_test_channel.type = file agent.channels.kafka_test_channel.checkpointdir =  /dev/shm/flume/checkpointsdir/auth agent.channels.kafka_test_channel.datadirs = /dev/shm/flume/datadir/auth agent.channels.kafka_test_channel.checkpointinterval = 10 agent.channels.kafka_test_channel.capacity = 10000 agent.channels.kafka_test_channel.transactioncapacity = 1000  agent.sinks.elasticsearchsink.type = elasticsearch agent.sinks.elasticsearchsink.hostnames = 10.42.242.78:9300 agent.sinks.elasticsearchsink.indexname = auth agent.sinks.elasticsearchsink.indextype = logs agent.sinks.elasticsearchsink.clustername = elasticsearch agent.sinks.elasticsearchsink.batchsize = 500 agent.sinks.elasticsearchsink.ttl = 5d agent.sinks.elasticsearchsink.serializer = org.apache.flume.sink.elasticsearch.elasticsearchdynamicserializer  agent.sinks.elasticsearchsink.channel = kafka_test_channel 

10.42.242.78 1 of publishing addresses have extracted logs, i.e.

publish_address {10.42.242.78:9300}, bound_addresses {10.42.242.78:9300}, {172.17.0.5:9300} 

on top of es, using elasticsearch-kopf, , have created index having following setup

{   "number_of_shards": "1",   "number_of_replicas": "1",   "mappings": {     "default": {       "properties": {         "uuid": {           "type": "string"         },         "brandid": {           "type": "string"         },         "creationdate": {           "type": "string"         }       }     }   } } 

flume connects publishing endpoints, not able see data in es, @ all. trying search using index, nothing comes up.

an example of message sent flume can be

{"uuid":"12345","brandid":"12345","creationdate":"2017-09-13t20:32:15.660","payload":"test"} 

what missing?


Comments

Popular posts from this blog

ZeroMQ on Windows, with Qt Creator -

unity3d - Unity SceneManager.LoadScene quits application -

python - Error while using APScheduler: 'NoneType' object has no attribute 'now' -