What are you looking for?
You can't optimize what you don't measure. Before tuning anything, establish baseline metrics for your queue:
Throughput — jobs processed per minute. Is it growing fast enough to keep up with your dispatch rate?
Wait time / Latency — how long from dispatch to execution? High latency = insufficient workers or wrong queue priority.
Queue depth — number of pending jobs. Growing queue = workers can't keep up.
Failure rate — percentage of jobs that fail. High failure rate = bugs or unreliable dependencies.
Memory per worker — how much RAM each worker uses. Determines how many workers you can run per server.
Job duration — average time to process one job. Slow jobs reduce overall throughput.
This is the most common question. The answer depends on your job profile.
Workers needed = (Jobs per minute) / (Jobs per worker per minute) Jobs per worker per minute = 60 / Average job duration in seconds Example: - 600 jobs/minute incoming - Average job takes 3 seconds - Jobs per worker per minute = 60 / 3 = 20 - Workers needed = 600 / 20 = 30 workers
IO-Bound jobs (sending emails, calling APIs, database queries) — workers spend most of their time waiting. You can run many workers per CPU core. A 4-core server can run 20–50 workers.
CPU-Bound jobs (image processing, PDF generation, data encryption) — workers actively use the CPU. Run no more than 1–2 workers per CPU core to avoid contention.
# Check memory usage of a running worker (Linux) ps aux | grep "queue:work" # Each Laravel worker typically uses 30–100MB at rest # After processing jobs, it can grow to 150–300MB depending on your code # Memory limit setting stops the worker before it gets too large: php artisan queue:work --memory=128
Redis is fast by default, but misconfiguration can severely limit queue throughput.
# .env — use different Redis DB numbers REDIS_DB=0 # queue driver uses DB 0 REDIS_CACHE_DB=1 # cache driver uses DB 1 # This prevents cache flush (artisan cache:clear) from wiping your queued jobs
By default, Redis is configured as a pure cache (data can be lost on restart). For queue jobs, you need persistence so jobs survive Redis restarts:
# /etc/redis/redis.conf # Enable AOF (Append-Only File) persistence for durability appendonly yes appendfsync everysec # fsync every second (good balance of performance vs safety) # Or use RDB snapshots (less safe but faster): save 900 1 # save after 900s if at least 1 key changed save 300 10 # save after 300s if at least 10 keys changed
If Redis runs out of memory, it needs to evict data. The wrong eviction policy can delete queue jobs. For queue-specific Redis instances, never use an eviction policy that removes arbitrary keys:
# /etc/redis/redis.conf — for dedicated queue Redis maxmemory-policy noeviction # Don't evict anything — return an error if memory is full # This lets you know when you're out of memory instead of silently losing jobs
When dispatching many jobs from one place (like a dispatch job), use Redis pipelining to batch the writes instead of one round-trip per job:
use Illuminate\Support\Facades\Redis;
// Manual Redis pipelining for bulk dispatch
Redis::pipeline(function ($pipe) use ($jobs) {
foreach ($jobs as $job) {
// This batches all LPUSH commands in one Redis round-trip
$pipe->lpush('queues:default', serialize($job));
}
});
// Note: In practice, Bus::batch() handles this more elegantly — use that instead
The database queue driver has known performance limitations, but they can be mitigated significantly.
The default migration creates a minimal index. For high-throughput scenarios, add composite indexes:
// In a new migration
Schema::table('jobs', function (Blueprint $table) {
// Speeds up the worker's "get next available job" query significantly
$table->index(['queue', 'reserved_at', 'available_at'], 'jobs_queue_lookup');
});
// The worker query looks like:
// SELECT * FROM jobs WHERE queue = ? AND reserved_at IS NULL AND available_at <= ?
// This index makes that query use an index scan instead of full table scan
# Prune failed jobs older than 48 hours (run weekly via scheduler)
php artisan queue:prune-failed --hours=48
// Add to scheduler
$schedule->command('queue:prune-failed --hours=72')->weekly();
For high-traffic apps using the database queue driver, use a separate database instance
for the jobs and failed_jobs tables.
This prevents queue polling from impacting your main application database:
# .env — dedicated queue database connection
QUEUE_DB_HOST=queue-db.internal
QUEUE_DB_DATABASE=app_queues
QUEUE_DB_USERNAME=queue_user
QUEUE_DB_PASSWORD=secret
# config/database.php
'queue_db' => [
'driver' => 'mysql',
'host' => env('QUEUE_DB_HOST', '127.0.0.1'),
'database' => env('QUEUE_DB_DATABASE', 'app_queues'),
'username' => env('QUEUE_DB_USERNAME', 'root'),
'password' => env('QUEUE_DB_PASSWORD', ''),
// ...
],
# config/queue.php — point database driver to this connection
'database' => [
'driver' => 'database',
'connection' => 'queue_db', // use dedicated queue database
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
],
Laravel Horizon can automatically scale the number of worker processes based on queue load. This is the most efficient way to handle variable traffic — fewer workers during quiet periods, more during peaks.
// config/horizon.php
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['payments', 'emails', 'default'],
'balance' => 'auto', // auto-scaling mode
'minProcesses' => 2, // always run at least 2 workers
'maxProcesses' => 20, // scale up to 20 workers under load
'balanceMaxShift' => 5, // scale by max 5 workers per check
'balanceCooldown' => 3, // check every 3 seconds
'memory' => 128,
'tries' => 3,
'timeout' => 60,
],
],
// Separate supervisor for heavy/slow jobs
'production-heavy' => [
'supervisor-2' => [
'connection' => 'redis',
'queue' => ['pdf', 'video', 'imports'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 5, // fewer workers — CPU intensive
'memory' => 512, // higher memory limit
'timeout' => 300, // longer timeout for heavy jobs
],
],
]
simple — divides workers evenly across queues. Predictable but doesn't adapt to load.
auto — allocates workers based on queue depth and wait time. Best for production.
false — no auto-scaling. Fixed worker count. Use for CPU-bound jobs where you don't want to over-commit.
When one server isn't enough, run queue workers across multiple servers. Laravel queues are designed for this — multiple workers can safely pull from the same queue.
Shared queue backend — all servers connect to the same Redis or database
Shared storage — files created by jobs must be on S3 or shared NFS, not local disk
Shared cache — unique job locks and queue:restart signals use the cache. Must be Redis, not file cache.
Shared session/broadcast — if jobs interact with sessions or broadcasting
# On Server 1 (via Supervisor) php artisan queue:work redis --queue=emails,default --sleep=3 --tries=3 # On Server 2 — same command, same Redis, they share the queue php artisan queue:work redis --queue=emails,default --sleep=3 --tries=3 # Both servers pull jobs from the same Redis list # Redis BRPOPLPUSH ensures each job is given to exactly one worker (atomic pop)
# Run on any server — the signal is stored in the shared cache # All workers on all servers will gracefully restart php artisan queue:restart
One slow job type can clog your entire queue. Identify and fix slow jobs before scaling.
// App\Jobs\Middleware\ProfileJobExecution.php
class ProfileJobExecution
{
public function handle(object $job, \Closure $next): void
{
$start = microtime(true);
$next($job);
$duration = round((microtime(true) - $start) * 1000, 2); // milliseconds
$jobName = class_basename($job);
// Log slow jobs (over 2 seconds)
if ($duration > 2000) {
\Log::warning("[Slow Job] {$jobName} took {$duration}ms", [
'job' => $jobName,
'duration' => $duration,
]);
}
// Send to metrics (Datadog, Prometheus, etc.)
app('metrics')->histogram('queue.job.duration', $duration, ['job' => $jobName]);
}
}
Horizon's dashboard shows per-job throughput and wait times. Use the "Metrics" tab to identify which job types are slowest, have the most failures, or have the longest wait times. These are your optimization targets.
N+1 queries — missing eager loading inside handle()
Missing database indexes — the job queries a large table without an index
Synchronous external API calls — no timeout, waiting for slow API responses
Large payload deserialization — serialized large objects take time to reconstruct
Memory pressure — worker near its memory limit, PHP spending time on garbage collection
Smaller job payloads mean faster serialization, faster queue writes, and less memory pressure. Here are concrete techniques:
// ❌ Large payload — entire collection passed to job
class ProcessOrderCollection implements ShouldQueue
{
public function __construct(protected Collection $orders) {}
// Serializing 1000 orders = megabytes of payload
}
// ✅ Small payload — just the IDs
class ProcessOrderCollection implements ShouldQueue
{
public function __construct(protected array $orderIds) {}
public function handle(): void
{
// Fetch in chunks to avoid memory issues
Order::whereIn('id', $this->orderIds)
->cursor()
->each(fn ($order) => $this->processOrder($order));
}
}
// ✅ Even better — dispatch individual jobs per order
// One job per order = independent retry per failure, no monolithic payload
Order::whereIn('id', $orderIds)->cursor()->each(function ($order) {
ProcessSingleOrder::dispatch($order->id);
});
Queue Infrastructure: ✓ Redis driver (not database) for high throughput ✓ Redis AOF persistence enabled ✓ Redis maxmemory-policy = noeviction ✓ Separate Redis DB for queue vs cache ✓ failed_jobs table indexed and pruned regularly Worker Configuration: ✓ numprocs calculated based on job duration and volume ✓ --max-jobs or --max-time set to prevent memory leaks ✓ --timeout set and aligned with job $timeout property ✓ --sleep=1 for high-priority queues, --sleep=5 for low-priority ✓ Separate workers for IO-bound vs CPU-bound jobs Job Code: ✓ Eager loading inside handle() — no lazy loading ✓ $deleteWhenMissingModels = true where appropriate ✓ Idempotent handle() — safe to run multiple times ✓ failed() method implemented on all critical jobs ✓ $afterCommit = true when dispatching near transactions ✓ Exponential backoff array on external API jobs ✓ Payload is IDs/references, not full models/collections Deployment: ✓ queue:restart in every deploy script ✓ Shared cache driver (Redis) for restart signals to work Monitoring: ✓ Horizon installed (Redis) OR queue:monitor scheduled (database) ✓ Alerts on failed job count exceeding threshold ✓ Alerts on queue depth exceeding threshold ✓ Slow job profiling in place
Queue performance is not a single lever — it's the combination of the right driver, the right worker configuration, well-written job code, and ongoing monitoring. A system that works at 100 jobs/day needs different tuning than one handling 1,000,000 jobs/day.
Start simple: measure first, then optimize the bottlenecks you find. Don't prematurely scale horizontally when the real problem is an N+1 query inside your jobs. Profile before you provision.
Switch from database to Redis driver before anything else — it's the biggest single improvement
Size workers based on your job duration profile, not guesswork
Use Horizon's auto-scaling to handle traffic spikes without over-provisioning
Keep payloads small — IDs and references only
Add a slow-job middleware to identify performance bottlenecks early
The checklist above is your pre-launch review for any production queue system