.Net HttpWebRequest.GetResponse() raises exception when http status code 400 (bad request) is returned

It would be nice if there were some way of turning off "throw on non-success code" but if you catch WebException you can at least use the response:

using System;
using System.IO;
using System.Web;
using System.Net;

public class Test
{
    static void Main()
    {
        WebRequest request = WebRequest.Create("http://csharpindepth.com/asd");
        try
        {
            using (WebResponse response = request.GetResponse())
            {
                Console.WriteLine("Won't get here");
            }
        }
        catch (WebException e)
        {
            using (WebResponse response = e.Response)
            {
                HttpWebResponse httpResponse = (HttpWebResponse) response;
                Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
                using (Stream data = response.GetResponseStream())
                using (var reader = new StreamReader(data))
                {
                    string text = reader.ReadToEnd();
                    Console.WriteLine(text);
                }
            }
        }
    }
}

You might like to encapsulate the "get me a response even if it's not a success code" bit in a separate method. (I'd suggest you still throw if there isn't a response, e.g. if you couldn't connect.)

If the error response may be large (which is unusual) you may want to tweak HttpWebRequest.DefaultMaximumErrorResponseLength to make sure you get the whole error.

I know this has already been answered a long time ago, but I made an extension method to hopefully help other people that come to this question.

Code:

public static class WebRequestExtensions
{
    public static WebResponse GetResponseWithoutException(this WebRequest request)
    {
        if (request == null)
        {
            throw new ArgumentNullException("request");
        }

        try
        {
            return request.GetResponse();
        }
        catch (WebException e)
        {
            if (e.Response == null)
            {
                throw;
            }

            return e.Response;
        }
    }
}

Usage:

var request = (HttpWebRequest)WebRequest.CreateHttp("http://invalidurl.com");

//... (initialize more fields)

using (var response = (HttpWebResponse)request.GetResponseWithoutException())
{
    Console.WriteLine("I got Http Status Code: {0}", response.StatusCode);
}

Interestingly, the HttpWebResponse.GetResponseStream() that you get from the WebException.Response is not the same as the response stream that you would have received from server. In our environment, we're losing actual server responses when a 400 HTTP status code is returned back to the client using the HttpWebRequest/HttpWebResponse objects. From what we've seen, the response stream associated with the WebException's HttpWebResponse is generated at the client and does not include any of the response body from the server. Very frustrating, as we want to message back to the client the reason for the bad request.

I had similar issues when trying to connect to Google's OAuth2 service.

I ended up writing the POST manually, not using WebRequest, like this:

TcpClient client = new TcpClient("accounts.google.com", 443);
Stream netStream = client.GetStream();
SslStream sslStream = new SslStream(netStream);
sslStream.AuthenticateAsClient("accounts.google.com");

{
    byte[] contentAsBytes = Encoding.ASCII.GetBytes(content.ToString());

    StringBuilder msg = new StringBuilder();
    msg.AppendLine("POST /o/oauth2/token HTTP/1.1");
    msg.AppendLine("Host: accounts.google.com");
    msg.AppendLine("Content-Type: application/x-www-form-urlencoded");
    msg.AppendLine("Content-Length: " + contentAsBytes.Length.ToString());
    msg.AppendLine("");
    Debug.WriteLine("Request");
    Debug.WriteLine(msg.ToString());
    Debug.WriteLine(content.ToString());

    byte[] headerAsBytes = Encoding.ASCII.GetBytes(msg.ToString());
    sslStream.Write(headerAsBytes);
    sslStream.Write(contentAsBytes);
}

Debug.WriteLine("Response");

StreamReader reader = new StreamReader(sslStream);
while (true)
{  // Print the response line by line to the debug stream for inspection.
    string line = reader.ReadLine();
    if (line == null) break;
    Debug.WriteLine(line);
}

The response that gets written to the response stream contains the specific error text that you're after.

In particular, my problem was that I was putting endlines between url-encoded data pieces. When I took them out, everything worked. You might be able to use a similar technique to connect to your service and read the actual response error text.

Try this (it's VB-Code :-):

Try

Catch exp As WebException
  Dim sResponse As String = New StreamReader(exp.Response.GetResponseStream()).ReadToEnd
End Try

An asynchronous version of extension function:

    public static async Task<WebResponse> GetResponseAsyncNoEx(this WebRequest request)
    {
        try
        {
            return await request.GetResponseAsync();
        }
        catch(WebException ex)
        {
            return ex.Response;
        }
    }

This solved it for me:
https://gist.github.com/beccasaurus/929007/a8f820b153a1cfdee3d06a9c0a1d7ebfced8bb77

TL;DR:
Problem:
localhost returns expected content, remote IP alters 400 content to "Bad Request"
Solution:
Adding <httpErrors existingResponse="PassThrough"></httpErrors> to web.config/configuration/system.webServer solved this for me; now all servers (local & remote) return the exact same content (generated by me) regardless of the IP address and/or HTTP code I return.