Overview:

We are requesting a native queueing system within the Rewst platform—similar in concept to AWS SQS—to support decoupled, event-based workflows with built-in failure handling, retry logic, and message buffering.

Current Capabilities and Limitations:

Rewst already supports:

Completion Handlers: Useful for chaining workflows, but tightly coupled and inflexible.

Webhooks: Effective for triggering workflows externally, but require explicit targeting.

Sub-Workflows: Helpful for reusability and modular design, but they execute synchronously and do not provide buffering, retry, or asynchronous decoupling.

These features are powerful but don't offer the resilience, queuing, and flow control needed for high-volume or failure-prone use cases.

--

Proposed Enhancement:

Introduce a first-class internal queue system that allows workflows to:

Push messages to a named queue, with an optional payload.

Trigger workflows based on messages in the queue, processing them individually or in batches.

Include visibility timeouts, retry attempts, and dead-letter queues for persistent failures.

Optionally support message delay, FIFO ordering, or filtering based on message content.

---

Example Use Cases:

A data sync workflow pushes jobs to a queue for downstream processing. If one job fails, it is retried automatically after a delay.

A webhook receives 1,000 requests in a short burst—these are queued and processed at a controlled rate by a consumer workflow.

A provisioning workflow emits tasks to a queue (provision.task) for systems like M365, Entra ID, or AWS, allowing consumers to act independently and retry failures without coordination.

---

Benefits:

Improved reliability: Work is not lost on failure—automated retries and DLQs provide resilience.

Better decoupling: Workflows don’t need to know who will consume their output.

Load smoothing: Handle event spikes without overloading the platform.

Simplified error handling: Avoid the need for complex retry loops or delay logic within workflows.

Reusability: Standardised queue triggers can be reused across workflows or accounts.

---

Optional Enhancements for Future Iteration:

Queue metrics and monitoring: e.g., message count, retry stats, error rates.

Manual requeueing or DLQ inspection in the UI.

Ability to publish messages from external systems into Rewst queues via webhook or API.