Send data from Apache Flume to ElasticSearch -

-

i using following flume 1.7 agent configuration stream data kafka 0.9.0.1 topic, , send data elasticsearch setup on rancher using es found in catalog version v0.5.0.

agent.sources  = flume_test agent.channels = kafka_test_channel agent.sinks =   elasticsearchsink  agent.sources.flume_test.type = org.apache.flume.source.kafka.kafkasource agent.sources.flume_test.zookeeperconnect = stage-kafka01.stage:2181,stage-kafka02.stage:2181,stage-kafka03.stage:2181 agent.sources.flume_test.topic = hrzn_stage1_test agent.sources.flume_test.groupid = flume_kafka_stage1_test agent.sources.flume_test.channels = kafka_test_channel agent.sources.flume_test.spooldir = /var/log/spooldir/auth agent.sources.flume_test.interceptors = i1 agent.sources.flume_test.interceptors.i1.type = com.cpm.interceptors.testinterceptor$builder  agent.channels.kafka_test_channel.type = file agent.channels.kafka_test_channel.checkpointdir =  /dev/shm/flume/checkpointsdir/auth agent.channels.kafka_test_channel.datadirs = /dev/shm/flume/datadir/auth agent.channels.kafka_test_channel.checkpointinterval = 10 agent.channels.kafka_test_channel.capacity = 10000 agent.channels.kafka_test_channel.transactioncapacity = 1000  agent.sinks.elasticsearchsink.type = elasticsearch agent.sinks.elasticsearchsink.hostnames = 10.42.242.78:9300 agent.sinks.elasticsearchsink.indexname = auth agent.sinks.elasticsearchsink.indextype = logs agent.sinks.elasticsearchsink.clustername = elasticsearch agent.sinks.elasticsearchsink.batchsize = 500 agent.sinks.elasticsearchsink.ttl = 5d agent.sinks.elasticsearchsink.serializer = org.apache.flume.sink.elasticsearch.elasticsearchdynamicserializer  agent.sinks.elasticsearchsink.channel = kafka_test_channel

10.42.242.78 1 of publishing addresses have extracted logs, i.e. 

publish_address {10.42.242.78:9300}, bound_addresses {10.42.242.78:9300}, {172.17.0.5:9300}

on top of es, using elasticsearch-kopf, , have created index having following setup 

{   "number_of_shards": "1",   "number_of_replicas": "1",   "mappings": {     "default": {       "properties": {         "uuid": {           "type": "string"         },         "brandid": {           "type": "string"         },         "creationdate": {           "type": "string"         }       }     }   } }

flume connects publishing endpoints, not able see data in es, @ all. trying search using index, nothing comes up.

an example of message sent flume can be

{"uuid":"12345","brandid":"12345","creationdate":"2017-09-13t20:32:15.660","payload":"test"}

what missing?


Comments

Post a Comment

Popular posts from this blog

ios - MKAnnotationView layer is not of expected type: MKLayer -

-
Image
so code works fine logger riddled message. there way rid of or suppress it? postannotation.swift class postannotation: mkpointannotation { //mark: properties let post: post //mark: initialization init(post: post) { self.post = post super.init() self.coordinate = cllocationcoordinate2d(latitude: post.latitude, longitude: post.longitude) self.title = post.title self.subtitle = post.timestring() } } adding annotation let annotation = postannotation(post: post) self.map.addannotation(annotation) func mapview func mapview(_ mapview: mkmapview, viewfor annotation: mkannotation) -> mkannotationview? { if annotation mkuserlocation { return nil } var annotationview = mapview.dequeuereusableannotationview(withidentifier: "pin") as? mkpinannotationview if annotationview == nil { annotationview = mkpinannotationview(annotation: annotation, reuseidentifier: "pin")
Read more

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more