Append data to HDF5 file with Pandas, Python -

-

i have large pandas dataframes financial data. have no problem appending , concatenating additional columns , dataframes .h5 file.

the financial data being updated every minute, need append row of data of existing tables inside of .h5 file every minute.

here have tried far, no matter do, overwrites .h5 file , not append data.

hdfstore way:

#we open hdf5 file save_hdf = hdfstore('test.h5')   ohlcv_candle.to_hdf('test.h5')  #we give dataframe key value #format=table can append data save_hdf.put('name_of_frame',ohlcv_candle, format='table',  data_columns=true)  #we print our dataframe calling hdf file key #just doing test print(save_hdf['name_of_frame'])

the other way have tried it, to_hdf: 

#format=t can append data , mode=r+ specify file exists , #we want append tohlcv_candle.to_hdf('test.h5',key='this_is_a_key', mode='r+', format='t')  #again printing check if worked  print(pd.read_hdf('test.h5', key='this_is_a_key'))

here 1 of dataframes looks after being read_hdf:

           time     open     high      low    close     volume           pp   0    1505305260  3137.89  3147.15  3121.17  3146.94   6.205397  3138.420000    1    1505305320  3146.86  3159.99  3130.00  3159.88   8.935962  3149.956667    2    1505305380  3159.96  3160.00  3159.37  3159.66   4.524017  3159.676667    3    1505305440  3159.66  3175.51  3151.08  3175.51   8.717610  3167.366667    4    1505305500  3175.25  3175.53  3170.44  3175.53   3.187453  3173.833333

the next time getting data (every minute), row of added index 5 of columns..and 6 , 7 ..and on, without having read , manipulate entire file in memory defeat point of doing this. if there better way of solving this, not shy recommend it.

p.s. sorry formatting of table in here

try this:

store = pd.hdfstore('test.h5')  store.append('name_of_frame', ohlcv_candle, format='t',  data_columns=true)

Comments

Post a Comment

Popular posts from this blog

ios - MKAnnotationView layer is not of expected type: MKLayer -

-
Image
so code works fine logger riddled message. there way rid of or suppress it? postannotation.swift class postannotation: mkpointannotation { //mark: properties let post: post //mark: initialization init(post: post) { self.post = post super.init() self.coordinate = cllocationcoordinate2d(latitude: post.latitude, longitude: post.longitude) self.title = post.title self.subtitle = post.timestring() } } adding annotation let annotation = postannotation(post: post) self.map.addannotation(annotation) func mapview func mapview(_ mapview: mkmapview, viewfor annotation: mkannotation) -> mkannotationview? { if annotation mkuserlocation { return nil } var annotationview = mapview.dequeuereusableannotationview(withidentifier: "pin") as? mkpinannotationview if annotationview == nil { annotationview = mkpinannotationview(annotation: annotation, reuseidentifier: "pin")
Read more

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more