In my case, I'm using the requests
library to call PayPal's API over HTTPS. Unfortunately, I'm getting an error from PayPal, and PayPal support cannot figure out what the error is or what's causing it. They want me to "Please provide the entire request, headers included".
How can I do that?
A simple method: enable logging in recent versions of Requests (1.x and higher.)
Requests uses the http.client
and logging
module configuration to control logging verbosity, as described here.
Demonstration
Code excerpted from the linked documentation:
import requests
import logging
# These two lines enable debugging at httplib level (requests->urllib3->http.client)
# You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA.
# The only thing missing will be the response.body which is not logged.
try:
import http.client as http_client
except ImportError:
# Python 2
import httplib as http_client
http_client.HTTPConnection.debuglevel = 1
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
requests.get('https://httpbin.org/headers')
Example Output
$ python requests-logging.py
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org
send: 'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-48-generic\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json
header: Date: Sat, 29 Jun 2013 11:19:34 GMT
header: Server: gunicorn/0.17.4
header: Content-Length: 226
header: Connection: keep-alive
DEBUG:requests.packages.urllib3.connectionpool:"GET /headers HTTP/1.1" 200 226
r = requests.get('https://api.github.com', auth=('user', 'pass'))
r
is a response. It has a request attribute which has the information you need.
r.request.allow_redirects r.request.headers r.request.register_hook
r.request.auth r.request.hooks r.request.response
r.request.cert r.request.method r.request.send
r.request.config r.request.params r.request.sent
r.request.cookies r.request.path_url r.request.session
r.request.data r.request.prefetch r.request.timeout
r.request.deregister_hook r.request.proxies r.request.url
r.request.files r.request.redirect r.request.verify
r.request.headers
gives the headers:
{'Accept': '*/*',
'Accept-Encoding': 'identity, deflate, compress, gzip',
'Authorization': u'Basic dXNlcjpwYXNz',
'User-Agent': 'python-requests/0.12.1'}
Then r.request.data
has the body as a mapping. You can convert this with urllib.urlencode
if they prefer:
import urllib
b = r.request.data
encoded_body = urllib.urlencode(b)
depending on the type of the response the .data
-attribute may be missing and a .body
-attribute be there instead.
response.request
seems to be a PreparedRequest
in my case; it doesn't have .data
but .body
instead.
response.url
(which is a bit different in that it's not response.request...
You can use HTTP Toolkit to do exactly this.
It's especially useful if you need to do this quickly, with no code changes: you can open a terminal from HTTP Toolkit, run any Python code from there as normal, and you'll be able to see the full content of every HTTP/HTTPS request immediately.
There's a free version that can do everything you need, and it's 100% open source.
I'm the creator of HTTP Toolkit; I actually built it myself to solve the exact same problem for me a while back! I too was trying to debug a payment integration, but their SDK didn't work, I couldn't tell why, and I needed to know what was actually going on to properly fix it. It's very frustrating, but being able to see the raw traffic really helps.
If you're using Python 2.x, try installing a urllib2 opener. That should print out your headers, although you may have to combine that with other openers you're using to hit the HTTPS.
import urllib2
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1)))
urllib2.urlopen(url)
A much simpler way to debug HTTP local requests is to use netcat. If you run
nc -l 1234
you'll start listening on port 1234
for HTTP connections. You can access it via http://localhost:1234/foo/foo/...
.
On the terminal, you'll see whatever raw data you sent to the endpoint. For example:
POST /foo/foo HTTP/1.1
Accept: application/json
Connection: keep-alive
Host: example.com
Accept-Language: en-en
Authorization: Bearer ay...
Content-Length: 15
Content-Type: application/json
{"test": false}
The verbose
configuration option might allow you to see what you want. There is an example in the documentation.
NOTE: Read the comments below: The verbose config options doesn't seem to be available anymore.
No logging system completely works, (as of requests 2.26 anyway, very old versions had maybe another behaviour)
The good solution is to use 'hooks' and print details as they happen.
This is fairly well explained here : https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/
under "printing everything",
but in case the link dies here is the important parts
import requests
from requests_toolbelt.utils import dump
def logging_hook(response, *args, **kwargs):
data = dump.dump_all(response)
print(data.decode('utf-8'))
http = requests.Session()
http.hooks["response"] = [logging_hook]
http.get("https://api.openaq.org/v1/cities", params={"country": "BA"})
The result this time will be a complete trace of sent query and received response.
I've tried it successfully with POST and lots of headers : it works. Don't forget to pip install requests_toolbelt.
# Output
< GET /v1/cities?country=BA HTTP/1.1
< Host: api.openaq.org
> HTTP/1.1 200 OK
> Content-Type: application/json; charset=utf-8
> Transfer-Encoding: chunked
> Connection: keep-alive
>
{
"meta":{
"name":"openaq-api",
"license":"CC BY 4.0",
"website":"https://docs.openaq.org/",
"page":1,
"limit":100,
"found":1
},
"results":[
{
"country":"BA",
"name":"Goražde",
"city":"Goražde",
"count":70797,
"locations":1
}
]
}
A previous answer seems to have been downvoted because it started with "nothing completely works" and then provides this perfect solution:
Install the requests_toolbelt collection of utilities with pip install requests-toolbelt. Use it like so: import requests from requests_toolbelt.utils import dump response = requests.get("https://v2.jokeapi.dev/joke/Any?safe-mode") print(dump.dump_all(response).decode("utf-8"))
Success story sharing
import httplib
withimport requests.packages.urllib3.connectionpool as httplib
or use six andfrom six.moves import http_client as httplib
.requests
2.18.1 and Python 3, the loggerlogging.getLogger("requests.packages.urllib3")
doesn't exist or has no effect.from http.client import HTTPConnection