A comparison of response times using URLLib, GRequests, and asyncio

For the HiPS client multiple tiles have to be fetched for time efficiency. To achieve this, we create a separate thread for each outgoing request. Thus, requests are sent concurrently. A comparison is done utilizing Python’s threading library. The elapsed time is calculated using the time module. For fetching the tiles urllib, grequests, aiohttp, and asyncio packages are used. The HiPS survey chosen for this comparison is alasky.u-strasbg.fr.

For fetching 10 tiles, it takes the following time (in seconds):

Elapsed Time URLLib (without concurrency): 3.5430831909179688
Elapsed Time URLLib (with concurrency): 0.388397216796875
Elapsed Time URLLib (with aiohttp): 0.3900480270385742
Elapsed Time GRequests: 1.6238431930541992

Similarly, for fetching 100 tiles, it takes:

Elapsed Time URLLib (without concurrency): 37.7027428150177
Elapsed Time URLLib (with concurrency): 5.575664043426514
Elapsed Time URLLib (with aiohttp): 2.4697625637054443
Elapsed Time GRequests: 4.273705244064331

The pros of grequests is that it takes less time when large number of requests have to be sent. But urllib (with threading) gives a better response time when requests are few.

Using aiohttp with asyncio seems to be the best option. Its response time is almost 50% less than grequests.

The source code for this comparison can be found here.