python - Understanding requests versus grequests -
i'm working process follows:
- build list/other iterable of urls.
- make http request each of them ,
response
object. - create beautifulsoup object
text
of each response. - pull text of tag beautifulsoup object.
from understanding, seems ideal grequests:
grequests allows use requests gevent make asynchronous http requests easily.
but yet, 2 processes (one requests, 1 grequests) seem getting me different results, of requests in grequests returning none
rather response.
using requests
import requests tickers=[ 'a', 'aal', 'aap', 'aapl', 'abbv', 'abc', 'abt', 'acn', 'adbe', 'adi', 'adm', 'adp', 'ads', 'adsk', 'aee', 'aep', 'aes', 'aet', 'afl', 'agn', 'aig', 'aiv', 'aiz', 'ajg', 'akam', 'alb', 'algn', 'alk', 'all', 'alle', ] base = 'https://finance.google.com/finance?q={}' # generator object @ point # understanding each function within generator not called until # list(rs) in following line. rs = (requests.get(u) u in [base.format(t) t in tickers]) rs = list(rs) rs # [<response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # ... # <response [200]>] # okay (status_code == 200)
using grequests
# restarted interpreter , redefined `tickers` , `base` import grequests # again, generator of unsent requests rs = (grequests.get(u) u in [base.format(t) t in tickers]) # send them @ same time rs = grequests.map(rs) rs # [none, # <response [200]>, # none, # none, # none, # none, # none, # none, # none, # none, # none, # none, # none, # none, # none, # none, # none, # none, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>, # <response [200]>]
why difference in results?
update: can print exception type follows. related discussion here have no idea what's going on.
def exception_handler(request, exception): print(exception) rs = grequests.map(rs, exception_handler=exception_handler) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",) # ("bad handshake: syscallerror(-1, 'unexpected eof')",)
Comments
Post a Comment