I assume this isn't a theoretical question, but an actual issue you encountered... within an hour of being used

I think this still has a race condition. At least the default implementation of Stream will only check before it calls BeginRead if the operation has been cancelled. Later cancel operations via the linked CancellatonTokenSource will not cause any side effect to the already running read operation e.g. a slow network read.

public virtual System.Threading.Tasks.Task<int> ReadAsync(byte[] buffer, int offset, int count, System.Threading.CancellationToken cancellationToken)
{
    if (!cancellationToken.IsCancellationRequested)
    {
        return this.BeginEndReadAsync(buffer, offset, count);
    }
    return System.Threading.Tasks.Task.FromCanceled<int>(cancellationToken);
}

The code above looks better but it will perform equally bad as the original version if the underlying stream was not overriden with another implementation. Cancelling is always asynchronous and you never know when you have reached a safe state except if you wait for the operation to complete and time out properly.

Yes NetworkStream seems to handle this properly. Thanks for the NetworkStream links. Async code is much harder to understand and riddled with race conditions. Personally I am still not convinced that the everything is async is the right thing to do. If you have chained thousands of async operations which work nicely in a good working system everything is fine but how do you handle in every layer a sudden slowdown because e.g. Network is slow or file operations take longer than before? In that case thousands of tasks pile up and will eventually hit the Threadpool Worker Thread limit leading to threadpool starvation and a sudden drop in throughput. If you have dependant tasks then you can even get a deadlock by awaiting tasks which are never scheduled to run because of the worker thread limit. That are hard to debug issues. You need to reverse engineer all of the task dependencies to get things working again which is no fun. Are there any libraries which make such things easier you would recommend?

Then you must define when you want to cancel. Is the system just slow or did it deadlock? How would you decide to cancel running operations? Cancel with a timeout as the code above implies? On a slow machine timing is very different compared to a big server machine. In principle it is easy but in practice it is still a hard problem.