apache spark - set outputCommitterClass property in hadoop2 -

-

i have been researching problem past few weeks, , didn't find clear answer.

here problem:

for hadoop1x (in mapred lib), use customized output committer using:

spark.conf.set(   "spark.hadoop.mapred.output.committer.class",   "some committer"  )

or calling jobconf.setoutputcommitter.

however, hadoop2x (in mapreduce lib), gets committer outputformat.getoutputcommitter, there no clear answer on how setoutputcommitter.

i found databricks set output committer using property, spark.hadoop.spark.sql.sources.outputcommitterclass.

i tried netflix's s3 committer(com.netflix.bdp.s3.s3directoryoutputcommitter), in log, spark still uses default committer:

17/09/13 22:39:36 info fileoutputcommitter: file output committer algorithm version 2 17/09/13 22:39:36 info directfileoutputcommitter: nothing clean since no temporary files written. 17/09/13 22:39:36 info csemultipartuploadoutputstream: close closed:false s3://xxxx/testtable3/.hive-staging_hive_2017-09-13_22-39-34_140_3769635956945982238-1/-ext-10000/_success

i'm wondering if it's possible overwrite default fileoutputcommitter , use customized committer in mapreduce lib?

how do it?

not no; it's i'm trying fix mapreduce-6823 -where you'll able set committer per filesystem schema. won't surface while (hadoop 3.1?)

you should able away setting sql output committer, though i'd check path. kicks in sql/dataframe work. can set parquet separately, though committer declare must subclass of parquetoutputcommitter, netflix 1 isn't.


Comments

Post a Comment

Popular posts from this blog

ios - MKAnnotationView layer is not of expected type: MKLayer -

-
Image
so code works fine logger riddled message. there way rid of or suppress it? postannotation.swift class postannotation: mkpointannotation { //mark: properties let post: post //mark: initialization init(post: post) { self.post = post super.init() self.coordinate = cllocationcoordinate2d(latitude: post.latitude, longitude: post.longitude) self.title = post.title self.subtitle = post.timestring() } } adding annotation let annotation = postannotation(post: post) self.map.addannotation(annotation) func mapview func mapview(_ mapview: mkmapview, viewfor annotation: mkannotation) -> mkannotationview? { if annotation mkuserlocation { return nil } var annotationview = mapview.dequeuereusableannotationview(withidentifier: "pin") as? mkpinannotationview if annotationview == nil { annotationview = mkpinannotationview(annotation: annotation, reuseidentifier: "pin")
Read more

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more