What are the differences between the urllib, urllib2, urllib3 and requests modules? - PyQuestions.com (2023)

I know it's been said before, but I highly recommend itRequestsPython-Package.

If you've used languages ​​other than Python, you're probably thinkingscreamingAndurllib2are easy to use, don't have much code and are very powerful, that's what I used to think. But theRequestsPack is so incredibly useful and short that everyone should use it.

First, it supports a fully restful API and is as simple as:

Import of requests resp = requests.get('http://www.mywebsite.com/user')resp = requests.post('http://www.mywebsite.com/user')resp = requests.put('http ://www.mywebsite.com/user/put')resp = requests.delete('http://www.mywebsite.com/user/delete')

Whether it's GET/POST, you never have to code parameters again, it just takes a dictionary as an argument and is good to go:

userdata = {"firstname": "John", "lastname": "Doe", "password": "jdoe123"}resp = requirements.post('http://www.mywebsite.com/user', data=userdata )

Plus, it even has a built-in JSON decoder (again, I knowjson.loads()there's not much more to write, but that's sure to come in handy):

bzw.json()

Or if your response data is just text, use:

or text

This is just the tip of the iceberg. This is the list of functions from the request side:

• International domains and URLs
• Keep-Alive & Connection Pooling
• Sessions with cookie persistence
• Browser-style SSL verification
• Basic/Digest authentication
• Elegant key/value cookies
• Automatic decompression
• Unicode response body
• Upload multipart files
• Connection Timeouts
• .netrc support
• list item
• Python 2.7, 3.6–3.9
• thread-locking.

urllib2 offers some additional functions, namely theVacation()function you can specify headers (typically in the past you would have had to use httplib, which is far more verbose). More importantly, urllib2 supports theInquiryclass that allows a more declarative approach to a request:

r = Request(url='http://www.mysite.com')r.add_header('User-Agent', 'awesome fetcher')r.add_data(urllib.urlencode({'foo': 'bar'} )Antwort = urlopen(r)

Note thatURL-Code ()is only in urllib, not urllib2.

There are also handlers to implement extended URL support in urllib2. The short answer is that unless you're working with legacy code, you probably want to use urllib2's url opener, but you still need to import into urllib for some of the helper functions.

Bonus answerWith Google App Engine, you can use httplib, urllib, or urllib2, but all of them are just wrappers for Google's URL fetching API. That means you're still subject to the same limitations, such as ports, protocols, and the length of the response allowed. However, you can use the core of the libraries to fetch HTTP URLs as expected.

This is my understanding of the relationships between the various "URLIBs":

In the Python 2 standard library, there are two HTTP libraries side by side. Despite the similar name, they are independent: they have a different design and implementation.

• screamingwas the original Python HTTP client added to the standard library inPython 1.2. Previous documentation forscreamingcan be found inPython 1.4.

• urllib2was a more powerful HTTP client,added in Python 1.6, intended as a replacement forscreaming:

  urllib2 - new and improved but incompatible version of urllib (still experimental).

  Previous documentation forurllib2can be found inPython 2.1.

The Python 3 standard library has oneneu screamingThis is a merged/redesigned/rewritten version of the older modules.

urllib3is a third-party package (i.e. not in CPython's standard library). Despite the name, it has nothing to do with the standard library packages, and there are no intentions to include it in the standard library in the future.

Finally,Requestsused internallyurllib3, but it aims for an easier to use API.

screamingAndurllib2are both Python modules related to URL requests but offer different functionalities.

1) urllib2 can accept a request object to set the headers for a url request, urllib only accepts a url.

2) urllib provides theURL-Codemethod used to generate GET query strings, urllib2 has no such function. This is one of the reasons why urllib is often used with urllib2.

Requests- Requests" is a simple, easy-to-use HTTP library written in Python.

1) Python Requests autocodes the parameters so you just pass them as simple arguments, unlike the case of urllib where you have to use the methodurllib.encode()to encode the parameters before passing them.

2) The response was automatically decoded into Unicode.

3) Requests also has much more convenient error handling. If your authentication fails, urllib2 would raise a urllib2.URLError while Requests would return a normal response object as expected. All you need to see if boolean's request was successfulAnswer.ok

Just to add to the existing answers, I don't see anyone mentioning that Python requirements are not a native library. If you're ok with adding dependencies, requirements are fine. However, if you're trying to avoid adding dependencies, urllib is a native Python library that's already available to you.

A major difference is the porting of Python2 to Python3. urllib2 does not exist for python3 and its methods ported to urllib. So if you use that a lot and plan to migrate to Python3 in the future, consider using urllib. However, the 2to3 tool automatically does most of the work for you.

I likeurllib.urlencodefunction and it doesn't seem to exist inurllib2.

>>> urllib.urlencode({'abc':'d f', 'def': '-!2'})'abc=d+f&def=-%212'

To get the content of a URL:

try: # Try importing requests first. import requestexcept ImportError: try: # Try to import Python3 urllib import urllib.request except AttributeError: # Now import Python2 urllib import urllibdef get_content(url): try: # Using requirements. return request.get(url).content # Returns "requests.models.Response". except NameError: try: # Using Python3 urllib. with urllib.request.urlopen(index_url) as response: return response.read() # Returns http.client.HTTPResponse. except AttributeError: # Using Python3 urllib. return urllib.urlopen(url).read() # Returns an instance.

It's hard to write Python2 and Python3 andInquiryDependency code for the answers because theyVacation()functions andrequests.get()Function returns different types:

• Python2urllib.request.urlopen()returns ahttp.client.HTTPResponse
• Python3urllib.urlopen(url)returns aExample
• Inquiryrequest.get(url)returns aInquiries.Models.Answer

I think all the answers are pretty good. But less details on urllib3. urllib3 is a very powerful HTTP client for Python. The following two commands work for the installation:

urllib3

my Pip,

pip install urllib3

or you can get the latest code from Github and install it by typing

$ git clone git://github.com/urllib3/urllib3.git$ cd urllib3$ install python setup.py

Then you are ready to go

Just import urllib3 with,

import urllib3

Instead of connecting directly, here you need a PoolManager instance to make requests. This does the connection pooling and thread safety for you. There is also a ProxyManager object for forwarding requests through an HTTP/HTTPS proxyYou can refer to the documentation here.Example usage:

>>> from urllib3 import PoolManager>>> manager = PoolManager(10)>>> r = manager.request('GET', 'http://google.com/')>>> r.headers['server' ]'gws'>>> r = manager.request('GET', 'http://yahoo.com/')>>> r.headers['server']'YTS/1.20.0'>>> r = manager.request('POST', 'http://google.com/mail')>>> r = manager.request('HEAD', 'http://google.com/calendar')>>> len (manager.pools)2>>> conn = manager.connection_from_host('google.com')>>> conn.num_requests3

As mentioned inurrlib3The documentation,urllib3brings many important features that are missing from the Python standard libraries.

• thread security.
• connection pooling.
• Client-side SSL/TLS inspection.
• File uploads with multipart encoding.
• Helpers for retrying requests and handling HTTP redirects.
• Support for gzip and deflate encoding.
• Proxy support for HTTP and SOCKS.
• 100% test coverage.

Follow the user manual for more details.

• response content(The HTTPResponse object provides status, data, and header attributes.)
• Using io wrappers with response content
• Creating a query parameter
• Extended use of urllib3

Requests

requests usedurllib3under the hood and make it even easier to makeRequestsand retrieve data. For one, keep-alive is automatic versus 100%urllib3where it isn't. It also has event hooks that invoke a callback function when an event is raised, such as: B. receiving a responseInRequests, each request type has its own function. So instead of creating a connection or pool, GET a URL directly.

To installRequestsjust run it with pip

Pip-Installationsanfragen

or you can just install from source,

$ git clone git://github.com/psf/requests.git$ cd-requests$ python setup.py install

Then,Import Requests

Here you can refer the officerDocumentation,More information about advanced usage like session object, SSL verification and event hooks can be found hereURL.

You should generally use urllib2 as it sometimes simplifies things by accepting request objects and also throws a URLException on protocol errors. However, with Google App Engine you cannot use both. You have to use thoseURL Fetch APIprovided by Google in its sandbox Python environment.

One key point I'm missing from the answers above is that urllib returns an object of type<Klasse http.client.HTTPResponse>whereasRequestsreturns<Class 'Requests.Models.Response'>.

For this reason the method read() can be usedscreamingbut not withRequests.

P.S. :Requestsis already rich in so many methods that hardly any more are needed thanread();>