json - Proces files fast in loop with function in R -

-

i have read 88 thousand json files , process them. r-script looks this:

webshop_id = numeric() serving = data.frame(value = numeric(), unit = character()) alcohol = data.frame(alcoholpercentage = numeric()) voeding = data.frame(character())  filenames <- list.files("d:/thijs/documents/ah/productfeatures/allproducts", pattern="*.json", full.names=true) tic() (i in 1:length(filenames)) {     df = fromjson(filenames[i])     webshop_id = c(webshop_id, df$product$webshopid)     if (is.null(df$productlabels$alcoholpercentage)==true){         alcohol = rbind(alcohol, alcoholpercentage = na)     } else{         alcohol = suppresswarnings(rbind(alcohol, alcoholpercentage = as.numeric(gsub("%","",df$productlabels$alcoholpercentage))))     }if (length(df$productlabels$nutrition$nutritionvalues[[1]])==0){         temp_row = data.frame(matrix(c(rep.int(na,length(voeding))),nrow=1,ncol=length(voeding)))         colnames(temp_row) = colnames(voeding)         voeding = rbind(voeding, temp_row)         serving = rbind(serving, data.frame(value = na, unit = na))     } else {         temp_row = suppresswarnings(t(as.character(sub(",",".",df$productlabels$nutrition$nutritionvalues[[1]]$description, fixed=true))))         colnames(temp_row) = t(df$productlabels$nutrition$nutritionvalues[[1]]$name)         temp_row = data.frame(temp_row)         voeding = rbind.fill(voeding,temp_row)         if (length(df$productlabels$nutrition$servingsizecode)==0){             serving = rbind(serving, data.frame(value = na, unit = na))         } else{             serving = rbind(serving, data.frame(value = as.numeric(df$productlabels$nutrition$servingsizevalue[1]),                                       unit = df$productlabels$nutrition$servingsizecode[1]))         }     } } toc()  nutrition = data.frame(webshop_id = webshop_id, serving, alcohol,  voeding)

when i'm running few files, there little no problems speed wise. however, 88 thousand, takes long time load in , process them. since lot of files have missing information, hence if functions.

is there more efficient way load in , process these files?


Comments

Post a Comment

Popular posts from this blog

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more

python - Error while using APScheduler: 'NoneType' object has no attribute 'now' -

-
i have python app hosting on heroku. use apscheduler call function every day @ time. strange thing 1 time function runs after while have following error log: exception in thread apscheduler (most raised during interpreter shutdown): 2017-09-06t15:03:28.347848+00:00 app[web.1]: traceback (most recent call last): 2017-09-06t15:03:28.347858+00:00 app[web.1]: file "/app/.heroku/python/lib/python2.7/threading.py", line 810, in __bootstrap_inner 2017-09-06t15:03:28.347859+00:00 app[web.1]: file "/app/.heroku/python/lib/python2.7/threading.py", line 763, in run 2017-09-06t15:03:28.347861+00:00 app[web.1]: file "/app/.heroku/python/lib/python2.7/site-packages/apscheduler/schedulers/blocking.py", line 27, in _main_loop 2017-09-06t15:03:28.348167+00:00 app[web.1]: <type 'exceptions.attributeerror'>: 'nonetype' object has no attribute 'now' any guesses? the full code long here key parts: def startdayauto(): global ...
Read more