Idempotent n8n Workflows: Concurrency, Locking, and Preventing Duplicate Side Effects

# What You’ll Learn#

Duplicates in automation are rarely a single bug. They’re usually an emergent behavior from retries, concurrent runs, and non-idempotent side effects like sending emails, charging cards, or creating CRM deals.

This guide explains how duplicates happen in real n8n setups, then gives production-ready patterns to make workflows idempotent under concurrency using dedupe keys, database locks, upserts, and the outbox pattern.

You’ll also get sample node setups and a final checklist you can use before going live.

# Why duplicates happen in n8n#

The fastest way to build reliability is to assume duplicates will happen and design for it. Distributed systems retry. Webhooks resend. Queues deliver at-least-once. Even “single-threaded” assumptions break when you scale n8n horizontally.

Below are the most common sources of duplicates, with concrete scenarios.

1) Retries at the workflow level and node level#

If a workflow fails after performing a side effect, a retry can repeat that side effect.

Example: you call a payment API, it succeeds, then the workflow crashes on a later node. On retry, you call the payment API again unless you store and check an idempotency key.

If you already use retries and alerting, keep going, but add idempotency. For patterns on safe retries and alarms, see n8n error handling, retries, and alerting.

2) Duplicate webhook deliveries#

Webhook providers commonly retry on timeout, 5xx responses, or connection errors. Many also retry when the response takes too long.

Typical behavior:

• provider sends webhook
• your workflow does 10 seconds of work
• provider timeout is 3 seconds
• provider retries the same webhook
• you now have two n8n executions running the same event

If you’re building webhook flows, see n8n webhook tutorial for endpoint setup and verification, then add dedupe keys and locks from this guide.

3) Parallel runs due to concurrency settings and horizontal scaling#

n8n can run multiple executions in parallel. That is good for throughput and bad for correctness if you depend on “only one at a time” behavior.

Parallel duplicates usually occur when:

• the same logical entity is processed in multiple executions at once
• your workflow reads state, computes a decision, then writes state
• two executions do the read before either writes

This creates race conditions like double-sending, double-updating, or violating business rules like “only one active subscription”.

4) Polling and data sync overlaps#

If you poll for changes on a schedule and the polling windows overlap, the same record can be pulled twice.

Example:

• you poll every 1 minute
• one run takes 90 seconds
• next run starts before the previous ends
• both runs fetch the same “last 5 minutes” of changes
• duplicates happen unless you dedupe

For deeper sync patterns, pagination, and dedup strategies, see n8n data sync, CDC, pagination, and deduplication.

ℹ️ Note: “Exactly-once” processing is rarely achievable end-to-end. Your goal is “effectively once” side effects using idempotency keys plus durable state.

# The core concept: separate events from side effects#

To make a workflow idempotent, you need a stable way to answer:

• Have I already processed this event
• Which side effects did I already perform
• Is a parallel execution currently processing it

That means you need:

1. 1
   A deterministic dedupe key per logical event
2. 2
   Durable storage for processing state, usually a database
3. 3
   Idempotent writes, typically via upserts or idempotency headers
4. 4
   Optional locks when concurrency must be serialized

# Choosing a dedupe key that actually works#

A dedupe key is only useful if it is stable and unique for the logical event.

Good sources for dedupe keys#

Rule of thumb#

• For webhooks, prefer provider event IDs.
• For data sync, prefer primary keys plus a version marker like updated timestamp.
• For internal triggers, create and persist a UUID at the source.

💡 Tip: If you can’t find a reliable event ID, create one by hashing a canonical subset of fields that define “same event”, then store that hash in your DB with a unique constraint.

# Pattern 1: Dedupe table with a unique constraint#

This is the simplest, most effective pattern for “don’t run side effects twice”.

Data model#

Create a table where the dedupe key is unique.

How it works#

1. 1
   On workflow start, attempt to insert dedupe_key.
2. 2
   If insert succeeds, you “own” processing and can continue.
3. 3
   If insert fails due to uniqueness, stop or short-circuit to “already processed”.

Sample n8n node setup#

Use PostgreSQL, MySQL, or any DB that supports unique constraints.

Step A: Compute the dedupe key

Use a Function node.

const payload = $json;
 
// Prefer a provider event id if present
const eventId = payload.id || payload.event_id;
 
// Fallback: stable composite key
const composite = [
  payload.account_id,
  payload.object_type,
  payload.object_id,
  payload.action,
  payload.updated_at,
].filter(Boolean).join(':');
 
const dedupeKey = eventId || composite;
 
return [{ dedupeKey }];

Step B: Insert the key

Use a Postgres node with an INSERT that does nothing on conflict.

INSERT INTO workflow_dedupe (dedupe_key, status, first_seen_at, last_seen_at)
VALUES ($1, 'processing', NOW(), NOW())
ON CONFLICT (dedupe_key) DO UPDATE
SET last_seen_at = NOW()
RETURNING status;

Then branch:

• If the row was newly inserted, continue.
• If it already existed and status is done, stop.
• If status is processing, decide whether to wait, stop, or treat as in-flight.

⚠️ Warning: A plain “lookup then insert” is not safe under concurrency. Two parallel executions can both see “not found” and both insert. Always use a single atomic insert with a unique constraint.

Step C: Mark done

At the end of the workflow, update status.

UPDATE workflow_dedupe
SET status = 'done', last_seen_at = NOW()
WHERE dedupe_key = $1;

When this pattern is enough#

Use it when:

• you mainly want to prevent duplicate external calls
• side effects can be safely skipped if already done
• you can tolerate “already processing” being dropped or handled manually

# Pattern 2: Idempotent upserts for all writes#

Many duplicates become harmless if every write is an upsert keyed by a stable ID.

Upsert examples#

• Creating a CRM contact should be “create or update by email”
• Syncing an order should be “upsert by order_id”
• Writing to your own DB should be “insert on conflict update”

Example: Upsert in Postgres node#

INSERT INTO orders (order_id, status, total_cents, updated_at)
VALUES ($1, $2, $3, NOW())
ON CONFLICT (order_id) DO UPDATE
SET status = EXCLUDED.status,
    total_cents = EXCLUDED.total_cents,
    updated_at = NOW();

Example: Upsert-like behavior in APIs#

Many SaaS APIs support “update if exists” by natural key:

• Contacts by email
• Users by external ID
• Products by SKU

If the API does not support upsert, you often implement a “search then create” flow, but that can race under concurrency. In that case:

• prefer a provider-side idempotency key header if available
• or add a DB lock around the search-create sequence

# Pattern 3: Provider idempotency keys for external APIs#

Some APIs provide a first-class idempotency mechanism. Stripe is the common example, but many payment, shipping, and invoicing providers have similar headers.

How to use in n8n#

• Compute a stable idempotency key per logical operation.
• Send it as a header in the HTTP Request node.

Example HTTP headers in the HTTP Request node:

• Idempotency-Key: your_dedupe_key

If your workflow retries the same request, the provider returns the same result instead of creating a second charge or a second shipment.

What to store#

Even with provider idempotency keys, store the mapping in your DB:

• dedupe key
• provider response ID
• status

This helps debugging and allows you to reconstruct state without querying the provider.

# Pattern 4: Database locking to serialize critical sections#

Locks are for cases where “two runs at the same time” is unacceptable, even if you dedupe. Typical examples:

• decrementing inventory
• allocating sequential invoice numbers
• enforcing “only one active subscription” transitions
• sending a single notification per account per day

Two types of locking you can use#

Example: Postgres advisory lock in n8n#

Use a Postgres node before the critical section:

SELECT pg_try_advisory_lock(hashtext($1)) AS locked;

If locked is false, you can:

• stop and rely on retry
• wait and retry after a delay
• return 202 and let the sender retry if it’s a webhook

After the critical section, release:

SELECT pg_advisory_unlock(hashtext($1)) AS unlocked;

Use a lock key like:

• account_id
• order_id
• invoice_series_id

🎯 Key Takeaway: Use locks to protect shared state transitions, not as a default anti-duplicate mechanism. Dedupe keys plus upserts should handle most workflows with lower operational risk.

# Pattern 5: Outbox pattern for reliable side effects#

The outbox pattern is the most robust approach when you need to update your DB and call an external API, and you cannot afford them to get out of sync.

The problem it solves#

If you do:

1. 1
   update DB
2. 2
   call external API

and step 2 fails after step 1 succeeded, retries may:

• call the external API twice
• or leave DB and external system inconsistent

If you do it in the opposite order, you can get the opposite inconsistency.

The outbox solution#

1. 1
   In a single DB transaction, write:

   • your state change
   • an “outbox event” row describing the external call to do
2. 2
   A separate workflow processes outbox rows:

   • sends external requests
   • marks outbox row as sent

This converts unreliable external calls into a durable queue you control.

Minimal outbox table#

Workflow design in n8n#

• Workflow A: receives webhook, validates, writes DB and outbox row, returns quickly.
• Workflow B: cron-triggered or queue-triggered, processes pending outbox rows, does external calls idempotently, marks sent.

This is especially effective for webhook senders with tight timeouts because Workflow A can respond in under 1 second.

# Sample workflow blueprint: webhook to invoice creation without duplicates#

This is a practical design that covers retries, duplicate webhooks, and concurrency.

Step 1: Webhook trigger#

Use Webhook node. Return response early after you have safely persisted the event.

If you need signature verification, do it before persisting.

Step 2: Generate a dedupe key#

Function node:

const body = $json.body || $json;
 
// Example: provider event id plus type
const dedupeKey = [body.event_id, body.type].filter(Boolean).join(':');
 
return [{ dedupeKey, body }];

Step 3: Atomic dedupe insert#

Postgres node insert into workflow_dedupe.

If duplicate:

• return HTTP 200 with “already processed” to stop provider retries
• or return 202 if you prefer provider retry semantics

Step 4: Write an outbox row#

Postgres node:

INSERT INTO outbox (id, event_type, payload, dedupe_key, status, created_at)
VALUES (gen_random_uuid(), 'invoice.create', $1, $2, 'pending', NOW())
ON CONFLICT (dedupe_key) DO NOTHING;

Store minimal payload:

• internal customer id
• amount
• currency
• invoice reference Avoid storing secrets.

Step 5: Respond to webhook#

Webhook Response node:

• status 200
• body “accepted”

Step 6: Outbox processor workflow#

Trigger: Cron every minute or a queue-like trigger if you have one.

Steps:

1. 1
   Select pending outbox rows with a limit, for example 50.
2. 2
   For each row, acquire a lock by dedupe_key.
3. 3
   Send HTTP Request to invoicing provider with idempotency header set to dedupe_key.
4. 4
   Mark row sent with provider response id.
5. 5
   Release lock.

# Concurrency patterns in n8n node terms#

n8n users often ask “Which node do I use for locking”. The answer is: use your database as the concurrency control plane.

Common node combinations#

# Handling “processing” state and stuck executions#

A dedupe row with status = processing can remain if an execution crashes.

Handle it explicitly:

• store started_at and updated_at
• treat “processing older than 15 minutes” as stale
• allow a new execution to take over by flipping status back to processing if stale

A safe takeover usually needs a lock or a compare-and-swap update.

Example takeover update:

UPDATE workflow_dedupe
SET status = 'processing', last_seen_at = NOW()
WHERE dedupe_key = $1
  AND status = 'processing'
  AND last_seen_at less than NOW() - INTERVAL '15 minutes';

Follow that with a check of affected rows to decide whether you now own processing.

# Production safety checklist for idempotent n8n workflows#

Use this as a go-live gate for any workflow that triggers side effects.

Event and dedupe#

• Every trigger defines a stable dedupe key.
• Dedupe key is stored in a DB table with a unique constraint.
• Workflow short-circuits on duplicates with a safe response, especially for webhooks.
• Dedupe records include timestamps and status for observability.

Concurrency and locking#

• Critical sections that touch shared state are protected by a lock keyed by entity ID.
• Lock acquisition failure is handled predictably, either retry with backoff or stop.
• Long-running executions do not hold locks longer than necessary.

Side effects and writes#

• All DB writes are upserts or otherwise idempotent.
• External APIs use provider idempotency headers where available.
• External create operations store the provider object ID to prevent re-creation.

Reliability and operations#

• Workflows respond to webhooks quickly after persisting state.
• Retries are enabled only for safe idempotent operations.
• Alerts exist for rising failure rates and repeated retries.
• A dead-letter path exists, such as marking outbox rows failed with an error reason.

For alerting patterns that catch repeat failures early, reference n8n error handling, retries, and alerting.

# Key Takeaways#

• Use a stable dedupe key per logical event and enforce uniqueness in a database to stop duplicates under retries and parallel runs.
• Prefer atomic operations like insert-on-conflict and upserts instead of “check then write” flows that race under concurrency.
• Add database locks only around true critical sections, like inventory, balances, or sequential numbering.
• For complex side effects, use the outbox pattern so DB state changes and external calls stay consistent across retries and crashes.
• Treat webhooks as at-least-once delivery, respond quickly after persisting state, and design for duplicate deliveries by default.
• Validate production readiness with a checklist covering dedupe, locking, idempotent writes, retries, and alerting.

# Conclusion#

Idempotency is the difference between an automation that “usually works” and one that survives real production conditions like retries, duplicate webhooks, and horizontal scaling. If you implement a dedupe key with a unique constraint, switch writes to upserts, and reserve locks for critical sections, you eliminate most duplicate side effects without slowing down delivery.

If you want Samioda to review your high-risk workflows, add an outbox architecture, or harden your n8n platform for concurrency at scale, contact us and we’ll help you ship automations that stay correct under load.