Queue Drivers

RouteMQ supports two queue drivers: Redis and Database. This guide explains how they work and when to use each.

Overview

Queue drivers handle the storage and retrieval of jobs. The driver you choose affects:

  • Performance - How fast jobs are queued and processed

  • Persistence - Whether jobs survive crashes

  • Infrastructure - What services you need to run

  • Scalability - How well it handles high job volumes

Redis Queue Driver

Fast, in-memory queue backed by Redis.

Features

  • Very fast - In-memory storage for low latency

  • High throughput - Handles thousands of jobs/second

  • Atomic operations - Uses RPOPLPUSH for safe job claiming

  • Delayed jobs - Efficient sorted sets for scheduling

  • Scalable - Easy to cluster Redis for more capacity

###Requirements

  • Redis server running (v5.0+)

  • ENABLE_REDIS=true in .env

  • QUEUE_CONNECTION=redis in .env

Configuration

Data Structures

Redis queue uses different data structures for different purposes:

Structure
Purpose
Type

routemq:queue:{name}

Pending jobs

List (FIFO)

routemq:queue:{name}:delayed

Delayed jobs

Sorted Set (by timestamp)

routemq:queue:{name}:reserved

Processing jobs

List

routemq:queue:failed:{name}

Failed jobs

List

How It Works

Pushing a job:

Popping a job:

Completing a job:

Failing a job:

Advantages

  • Speed: Sub-millisecond latency

  • Throughput: Handle high job volumes

  • Simple: No complex queries needed

  • Scalable: Easy to add Redis replicas

Disadvantages

  • Volatility: Jobs lost if Redis crashes (unless AOF enabled)

  • Infrastructure: Requires Redis server

  • Memory: All jobs stored in RAM

Best For

  • High-volume applications

  • Real-time job processing

  • When you already use Redis

  • When speed is critical

Database Queue Driver

Persistent queue backed by MySQL.

Features

  • Persistent - Jobs survive crashes

  • ACID - Transactional guarantees

  • Inspectable - Easy to query with SQL

  • Reliable - No job loss

  • No extra service - Uses existing MySQL

Requirements

  • MySQL server running (v8.0+)

  • ENABLE_MYSQL=true in .env

  • QUEUE_CONNECTION=database in .env

Configuration

Database Tables

queue_jobs:

Column
Type
Description

id

INT

Primary key

queue

VARCHAR(255)

Queue name

payload

TEXT

Serialized job data

attempts

INT

Number of attempts

reserved_at

DATETIME

When job was claimed (NULL if pending)

available_at

DATETIME

When job becomes available

created_at

DATETIME

When job was created

queue_failed_jobs:

Column
Type
Description

id

INT

Primary key

connection

VARCHAR(255)

Connection name

queue

VARCHAR(255)

Queue name

payload

TEXT

Serialized job data

exception

TEXT

Exception details

failed_at

DATETIME

When job failed

How It Works

Pushing a job:

Popping a job:

Completing a job:

Failing a job:

Advantages

  • Persistence: Jobs survive crashes

  • Reliability: ACID transactions

  • Visibility: Easy to inspect with SQL

  • Simplicity: No additional infrastructure

Disadvantages

  • Speed: Slower than Redis (disk I/O)

  • Load: Increases database queries

  • Scaling: Harder to scale than Redis

Best For

  • Low to medium job volumes

  • When persistence is critical

  • When you don't want to manage Redis

  • When you need to inspect jobs easily

Comparison

Feature
Redis
Database

Speed

⭐⭐⭐⭐⭐ Very Fast

⭐⭐⭐ Moderate

Persistence

⭐⭐ Configurable

⭐⭐⭐⭐⭐ Always

Scalability

⭐⭐⭐⭐⭐ Excellent

⭐⭐⭐ Good

Reliability

⭐⭐⭐ Good

⭐⭐⭐⭐⭐ Excellent

Setup

⭐⭐⭐ Need Redis

⭐⭐⭐⭐⭐ Use existing DB

Inspection

⭐⭐⭐ Redis CLI

⭐⭐⭐⭐⭐ SQL queries

Memory

⭐⭐ All in RAM

⭐⭐⭐⭐⭐ On disk

Switching Drivers

You can switch drivers at any time:

Important notes:

  • Jobs in the old driver won't be transferred

  • Complete existing jobs before switching

  • Or manually migrate jobs between drivers

Performance Tips

Redis

Database

Monitoring

Redis

Database

Troubleshooting

Redis Connection Issues

Database Connection Issues

Jobs Not Processing

  1. Check driver configuration:

  2. Verify service is running:

  3. Check worker connection:

Best Practices

1. Choose Based on Requirements

2. Monitor Both

Even if using Redis, keep failed jobs in database:

3. Regular Cleanup

4. Backup Important Queues

Next Steps

Last updated