multithreading - Python & web scraping performance -

-

i trying python based web scraping execution time pretty critical.

i've tried phantomjs, selenium, , pyqt4 now, , 3 libraries have given me similar response times. i'd post example code, problem affects three, believe problem either lies in shared dependency or outside of code. @ around 50 concurrent requests, see huge desegregation in response time. takes 40 seconds 50 pages, , time gets exponentially slower greater page demands. ideally i'm looking ~200+ requests in 10 seconds. used multiprocessing spawn each instance of phantonjs/pyqt4/selenium, each url request gets it's own instance i'm not blocked single threading.

i don't believe it's hardware bottleneck, it's running on 32 dedicated cpu cores, totaling 64 threads, , cpu usage doesn't typically spike on 10-12%. bandwidth sits reasonably comfortably @ around 40-50% of total throughput.

i've read gil, believe i've addressed using multiprocessing. webscraping inherently slow thing? should stop expecting pull 200ish webpages in ~10 seconds?

my overall question is, best approach high performance web scraping, evaluating js on webpage requirement?

"evaluating js on webpage requirement" <- think problem right here. downloading 50 web pages trivially parallelized , should take long slowest server takes respond. now, spawning 50 javascript engines in parallel (which guess doing) run scripts on every page different matter. imagine firing 50 chrome browsers @ same time.

anyway: profile , measure parts of application find bottleneck lies. can see if you're dealing i/o bottleneck (sounds unlikely), cpu bottleneck (more likely) or global lock somewhere serializes stuff (also impossible without code posted)


Comments

Post a Comment

Popular posts from this blog

ios - MKAnnotationView layer is not of expected type: MKLayer -

-
Image
so code works fine logger riddled message. there way rid of or suppress it? postannotation.swift class postannotation: mkpointannotation { //mark: properties let post: post //mark: initialization init(post: post) { self.post = post super.init() self.coordinate = cllocationcoordinate2d(latitude: post.latitude, longitude: post.longitude) self.title = post.title self.subtitle = post.timestring() } } adding annotation let annotation = postannotation(post: post) self.map.addannotation(annotation) func mapview func mapview(_ mapview: mkmapview, viewfor annotation: mkannotation) -> mkannotationview? { if annotation mkuserlocation { return nil } var annotationview = mapview.dequeuereusableannotationview(withidentifier: "pin") as? mkpinannotationview if annotationview == nil { annotationview = mkpinannotationview(annotation: annotation, reuseidentifier: "pin")
Read more

ZeroMQ on Windows, with Qt Creator -

-
i have installed zmq link given below on windows 7. http://zeromq.org/distro:microsoft-windows 1) have c++ project in qt-creator , i'm using zmq c++ binding (zhelpers.hpp etc). 2) have added .lib installed zmq folder qtcreator 3) while compiling, compiler looks _imp__zmq_verion, in .dll files present in .lib have __imp_zmq_version. 4) causing undefined reference zmq_version. anyone else faced problem , has solution ?
Read more

unity3d - Unity SceneManager.LoadScene quits application -

-
i have 2 different scenes 1 named 'main' , other 'game' both added scenes in load. when use scenemanager.loadscene("game"); works no problem when tried return main scene code if (input.getkeydown (keycode.escape)) { scenemanager.loadscene("main"); } strangely game quits. thanks. sorry interruption. somewhere on other script found code. if (input.getkeydown (keycode.escape)) { application.quit (); } that's bad, know...
Read more