I see, thanks. But even if we need a separate sync object to mark async complete completion, the server should have everything needed to know when to signal it, including the result of system APC which should deliver IOSB status. Do we really need a distinct server call to deliver notification for async completion on the client?
No, we don't, and I apologize, I should have noticed and replied to that point as well.
The server is made aware of async completion after the IOSB (and output buffers, if applicable) are filled. That's either done by waiting on the async handle itself (synchronous completion for old-style client-driven I/O, as currently used by serials), by calling set_async_direct_result (synchronous completion for new-style client-driven I/O, as currently used by sockets), or by closing the APC_ASYNC_IO thread APC handle by returning back to select (asynchronous completion).
This is necessary because completion has to be signaled (event, IOCP, user APC) after the IOSB and buffers are filled. All of this is done in async_set_result(); cancellation can also be handled there.
It's probably worth also noting, as I look at the code, that checking for STATUS_CANCELLED is neither sufficient (cancelled asyncs can still continue as normal for various reasons) nor necessary (STATUS_CANCELLED can be returned for asyncs that are not formally cancelled with NtCancelIo etc.)