python - Understanding requests versus grequests -


i'm working process follows:

  1. build list/other iterable of urls.
  2. make http request each of them , response object.
  3. create beautifulsoup object text of each response.
  4. pull text of tag beautifulsoup object.

from understanding, seems ideal grequests:

grequests allows use requests gevent make asynchronous http requests easily.

but yet, 2 processes (one requests, 1 grequests) seem getting me different results, of requests in grequests returning none rather response.

using requests

import requests  tickers=[     'a', 'aal', 'aap', 'aapl', 'abbv', 'abc', 'abt', 'acn', 'adbe', 'adi',      'adm',  'adp', 'ads', 'adsk', 'aee', 'aep', 'aes', 'aet', 'afl', 'agn',      'aig', 'aiv', 'aiz', 'ajg', 'akam', 'alb', 'algn', 'alk', 'all', 'alle',     ]  base = 'https://finance.google.com/finance?q={}'  # generator object @ point # understanding each function within generator not called until #     list(rs) in following line. rs = (requests.get(u) u in [base.format(t) t in tickers]) rs = list(rs)  rs # [<response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # ...  # <response [200]>]  # okay (status_code == 200) 

using grequests

# restarted interpreter , redefined `tickers` , `base` import grequests  # again, generator of unsent requests rs = (grequests.get(u) u in [base.format(t) t in tickers])  # send them @ same time rs = grequests.map(rs)  rs # [none,  # <response [200]>,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # none,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>,  # <response [200]>] 

why difference in results?

update: can print exception type follows. related discussion here have no idea what's going on.

def exception_handler(request, exception):     print(exception)  rs = grequests.map(rs, exception_handler=exception_handler)  # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) 


Comments

Popular posts from this blog

ZeroMQ on Windows, with Qt Creator -

unity3d - Unity SceneManager.LoadScene quits application -

python - Error while using APScheduler: 'NoneType' object has no attribute 'now' -