2.3.3 Requests and completion callback nesting

Most device driver class APIs are asynchronous by design. They work by submitting some kind of request to the driver which are processed asynchronously. The driver function used to submit the request always returns immediately and a callback mechanism is used to notify the caller that the request has been processed. The caller has to provide a completion handler function.

The callback mechanism used is in fact the kernel deferred routine service defined in mutek/kroutine.h. The actual execution of the completion handler may occur at various times depending on both the driver used and the policy the caller as chosen for the struct kroutine_s object of the submitted request.

struct device_timer_s timer_dev;
struct dev_timer_rq_s timer_rq;

static KROUTINE_EXEC(timer_event)
{
  printk("timer deadline!\n");
}

void timer_start()
{
  dev_timer_rq_init_immediate(&timer_rq, timer_event);
  ... DEVICE_OP(&timer_dev, request, &timer_rq);
}

Basically the policy may require immediate or deferred execution of the completion handler. Depending on the policy, it is executed or scheduled when the kroutine_exec function is called. The driver may actually invoke this function from different execution contexts when the request processing terminates:

The kroutine_exec function may be called nested from the driver function used to submit the request. In this case, the request processing terminates even before the request submitting function returns to the caller.
The kroutine_exec function may be called from an irq handler.
The kroutine_exec function may be called from an interruptible context when the driver internally uses a deferred processing mechanism.
The kroutine_exec function may be even called from an other completion handler when an other driver is used internally by the driver.

The execution context may vary between requests. For instance, a driver may choose to use a nested call when an error needs to be reported immediately or when the amount of requested data is already available from its internal buffers.

On the other side, the request submitter may want to start a new request as soon as a previous request completes. It will then call the request function of the driver from the completion handler.

There is a risk of deep function calls nesting when the following conditions are all met:

The driver calls the kroutine_exec function nested in its request submitting function (1),
the policy of the struct kroutine_s object chosen by the caller is KROUTINE_IMMEDIATE (2) and
the caller submits a new request from its completion handler function (3).

In order to avoid this issue, some rules must be followed so one of the above condition does not occur. When (1) is not allowed, the driver can not report an immediate error through the completion handler and needs to make the submit function return an error code. This creates two different code path for error handling by the caller and might as well prevent code factoring in the driver. This is also an issue in case the driver is able to satisfy the request immediately without waiting for an irq.

On the other hand, when (2) and (3) are not allowed to occur simultaneously, the submission of a new request by the caller will incur a slightly higher latency.

The retained rules actually depend on the class of the driver API.

2.3.3.1 Nested device request completion [link]

Most driver class APIs allow the driver to perform a nested call to the kroutine_exec function from their request submitting function. In this case, the request submitter is not allowed to submit an other request of any kind from its completion handler when the policy is KROUTINE_IMMEDIATE or similar.

This policy must be used only when the completion handler is used to perform a short task like waking a scheduler context. In most other cases, it should use the KROUTINE_DEFERRED policy for its completion handler. The handler will then execute from an interruptible context and will not be nested in any other call.

When this approach is used, the request submitting function of the driver API does not return a value and all possible errors are reported through the request completion handler.

Note that relying on deferred processing is fair because it still allows good execution latency provided that preemptive context switch is used along with the scheduler priority feature. Performing a long processing in a completion handler with immediate policy may prevent an urgent task from executing. Relying on deferred execution is a way to let the scheduler decide what is urgent.

struct device_char char_dev;
struct dev_char_rq_s char_rq;

static KROUTINE_EXEC(char_read_done)
{
  DEVICE_OP(&char_dev, request, &char_rq);
  ...
}

void start()
{
  dev_timer_rq_init(&char_rq, char_read_done);
  DEVICE_OP(&char_dev, request, &char_rq);
  ...
}

2.3.3.2 Nested device request submission [link]

Some driver class APIs like Timers and DMAs require short latency. In this case, the request submitter is allowed to submit an request to such a driver from a completion handler even when the policy is KROUTINE_IMMEDIATE.

The driver in turn must report any immediate error by making its request submitting function return an error code. In this case the request submission is rejected and the completion handler is never called.

When no error is returned, the request is not allowed to terminate immediately. The driver must defer the call to the kroutine_exec function by relying on an asynchronous event. This include relying on an irq, a deferred request to an other driver or a self scheduled deferred kroutine.

struct device_timer_s timer_dev;
struct dev_timer_rq_s timer_rq;

static KROUTINE_EXEC(timer_event)
{
  error_t err = DEVICE_OP(&timer_dev, request, &timer_rq);
  ...
}

void start()
{
  dev_timer_rq_init_immediate(&timer_rq, timer_event);
  error_t err = DEVICE_OP(&timer_dev, request, &timer_rq);
  ...
}

Request cancellation [link]

When a request cancellation function is provided by the driver, the function return value indicates if the request has been canceled immediately. In this case the function returns 0 and the completion handler is never invoked.

When this is not possible, the driver will terminate the request as soon as possible and the completion handler will be invoked as usual.

In the user code, canceling a request may be performed from a critical section in order to avoid race conditions due to the two possible code paths. For that reason, the driver must never invoke the completion handler directly from its cancel function.