Why would one use Task<T> over ValueTask<T> in C#?

From the API docs (emphasis added):

Methods may return an instance of this value type when it's likely that the result of their operations will be available synchronously and when the method is expected to be invoked so frequently that the cost of allocating a new Task for each call will be prohibitive. There are tradeoffs to using a ValueTask instead of a Task. For example, while a ValueTask can help avoid an allocation in the case where the successful result is available synchronously, it also contains two fields whereas a Task as a reference type is a single field. This means that a method call ends up returning two fields worth of data instead of one, which is more data to copy. It also means that if a method that returns one of these is awaited within an async method, the state machine for that async method will be larger due to needing to store the struct that's two fields instead of a single reference. Further, for uses other than consuming the result of an asynchronous operation via await, ValueTask can lead to a more convoluted programming model, which can in turn actually lead to more allocations. For example, consider a method that could return either a Task with a cached task as a common result or a ValueTask. If the consumer of the result wants to use it as a Task, such as to use with in methods like Task.WhenAll and Task.WhenAny, the ValueTask would first need to be converted into a Task using AsTask, which leads to an allocation that would have been avoided if a cached Task had been used in the first place. As such, the default choice for any asynchronous method should be to return a Task or Task. Only if performance analysis proves it worthwhile should a ValueTask be used instead of Task.

However I still do not understand what is the problem with using ValueTask always

Struct types are not free. Copying structs that are larger than the size of a reference can be slower than copying a reference. Storing structs that are larger than a reference takes more memory than storing a reference. Structs that are larger than 64 bits might not be enregistered when a reference could be enregistered. The benefits of lower collection pressure may not exceed the costs.

Performance problems should be approached with an engineering discipline. Make goals, measure your progress against goals, and then decide how to modify the program if goals are not met, measuring along the way to make sure that your changes are actually improvements.

why async/await wasn't built with a value type from the start.

await was added to C# long after the Task<T> type already existed. It would have been somewhat perverse to invent a new type when one already existed. And await went through a great many design iterations before settling on the one that was shipped in 2012. The perfect is the enemy of the good; better to ship a solution that works well with the existing infrastructure and then if there is user demand, provide improvements later.

I note also that the new feature of allowing user-supplied types to be the output of a compiler-generated method adds considerable risk and testing burden. When the only things you can return are void or a task, the testing team does not have to consider any scenario in which some absolutely crazy type is returned. Testing a compiler means figuring out not just what programs people are likely to write, but what programs are possible to write, because we want the compiler to compile all legal programs, not just all sensible programs. That's expensive.

Can someone explain when ValueTask would fail to do the job?

The purpose of the thing is improved performance. It doesn't do the job if it doesn't measurably and significantly improve performance. There is no guarantee that it will.

There is some changes in .Net Core 2.1. Starting from .net core 2.1 ValueTask can represent not only the synchronous completed actions but the async completed too. In addition we receive non-generic ValueTask type.

I will leave Stephen Toub comment which is related to your question:

We still need to formalize guidance, but I expect it'll be something like this for public API surface area: Task provides the most usability. ValueTask provides the most options for performance optimization. If you're writing an interface / virtual method that others will override, ValueTask is the right default choice. If you expect the API to be used on hot paths where allocations will matter, ValueTask is a good choice. Otherwise, where performance isn't critical, default to Task, as it provides better guarantees and usability. From an implementation perspective, many of the returned ValueTask instances will still be backed by Task.

Feature can be used not only in the .net core 2.1. You will be able to use it with System.Threading.Tasks.Extensions package.

A more recent info from Marc (Aug 2019)

Use Task when something is usually or always going to be genuinely asynchronous, i.e. not immediately complete; use ValueTask when something is usually or always going to be synchronous, i.e. the value will be known inline; also use ValueTask in a polymorphic scenario (virtual, interface) where you can't know the answer.

Source: https://blog.marcgravell.com/2019/08/prefer-valuetask-to-task-always-and.html

I followed above blog post for a recent project when I had similar questions.