Struct tower_batch_control::service::Batch

pub struct Batch<T, Request>
where
    T: Service<BatchControl<Request>>,
{
    tx: UnboundedSender<Message<Request, T::Future>>,
    semaphore: PollSemaphore,
    permit: Option<OwnedSemaphorePermit>,
    error_handle: ErrorHandle,
    worker_handle: Arc<Mutex<Option<JoinHandle<()>>>>,
}

Allows batch processing of requests.

See the crate documentation for more details.

Fields

tx: UnboundedSender<Message<Request, T::Future>>

A custom-bounded channel for sending requests to the batch worker.

Note: this actually is bounded, but rather than using Tokio’s unbounded channel, we use tokio’s semaphore separately to implement the bound.

semaphore: PollSemaphore

A semaphore used to bound the channel.

When the buffer’s channel is full, we want to exert backpressure in poll_ready, so that callers such as load balancers could choose to call another service rather than waiting for buffer capacity.

Unfortunately, this can’t be done easily using Tokio’s bounded MPSC channel, because it doesn’t wake pending tasks on close. Therefore, we implement our own bounded MPSC on top of the unbounded channel, using a semaphore to limit how many items are in the channel.

permit: Option<OwnedSemaphorePermit>

A semaphore permit that allows this service to send one message on tx.

error_handle: ErrorHandle

An error handle shared between all service clones for the same worker.

worker_handle: Arc<Mutex<Option<JoinHandle<()>>>>

A worker task handle shared between all service clones for the same worker.

Only used when the worker is spawned on the tokio runtime.

Implementations

impl<T, Request> Batch<T, Request>

where T: Service<BatchControl<Request>>, T::Future: Send + 'static, T::Error: Into<Box<dyn Error + Send + Sync + 'static>>,

pub fn new( service: T, max_items_in_batch: usize, max_batches: impl Into<Option<usize>>, max_latency: Duration, ) -> Self

where T: Send + 'static, T::Future: Send, T::Response: Send, T::Error: Send + Sync, Request: Send + 'static,

Creates a new Batch wrapping service.

The wrapper is responsible for telling the inner service when to flush a batch of requests. These parameters control this policy:

• max_items_in_batch gives the maximum number of items per batch.
• max_batches is an upper bound on the number of batches in the queue, and the number of concurrently executing batches. If this is None, we use the current number of [rayon] threads. The number of batches in the queue is also limited by QUEUE_BATCH_LIMIT.
• max_latency gives the maximum latency for a batch item to start verifying.

The default Tokio executor is used to run the given service, which means that this method must be called while on the Tokio runtime.

pub fn pair( service: T, max_items_in_batch: usize, max_batches: impl Into<Option<usize>>, max_latency: Duration, ) -> (Self, Worker<T, Request>)

where T: Send + 'static, T::Error: Send + Sync, Request: Send + 'static,

Creates a new Batch wrapping service, but returns the background worker.

This is useful if you do not want to spawn directly onto the tokio runtime but instead want to use your own executor. This will return the Batch and the background Worker that you can then spawn.

pub fn register_worker(&mut self, worker_handle: JoinHandle<()>)

Ask the Batch to monitor the spawned worker task’s [JoinHandle].

Only used when the task is spawned on the tokio runtime.

fn get_worker_error(&self) -> Box<dyn Error + Send + Sync + 'static>

Returns the error from the batch worker’s error_handle.

Trait Implementations

impl<T, Request> Clone for Batch<T, Request>

where T: Service<BatchControl<Request>>,

fn clone(&self) -> Self

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T, Request> Debug for Batch<T, Request>

where T: Service<BatchControl<Request>>,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T, Request> Service<Request> for Batch<T, Request>

where T: Service<BatchControl<Request>>, T::Future: Send + 'static, T::Error: Into<Box<dyn Error + Send + Sync + 'static>>,

type Response = <T as Service<BatchControl<Request>>>::Response

Responses given by the service.

type Error = Box<dyn Error + Sync + Send>

Errors produced by the service.

type Future = ResponseFuture<<T as Service<BatchControl<Request>>>::Future>

The future response value.

fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>>

Returns Poll::Ready(Ok(())) when the service is able to process requests.

fn call(&mut self, request: Request) -> Self::Future

Process the request and return the response asynchronously.