Claim an asyncio Job Before Awaiting: Reproduce and Fix a Duplicate Worker
Reproduce two asyncio workers claiming one job, protect the state transition with a shared lock, and verify the correction without claiming distributed exactly-once execution.

When two concurrent tasks invoke an uncoordinated claim routine against a list containing a single pending dictionary, the execution returns the duplicate list [1, 1]. In the synthetic reproducer, both worker tasks evaluate the condition checking whether a record is pending while the dictionary remains in its initial state. Immediately following that check, the routine reaches an explicit suspension point where control yields back to the event loop before the mutation occurs. In the verified test suite, this suspension is represented by an awaited zero-second sleep. Because the first worker suspends execution after validating the pending status but prior to writing the running status, the event loop schedules the second worker. That second worker evaluates the exact same dictionary, observes the pending status still intact, suspends at the identical yield point, and subsequently both workers execute their state assignments and return job identifier 1.
Two workers can observe the same pending state
Developers frequently assume that because asyncio executes on a single operating system thread, access to shared Python objects is naturally protected from race conditions. While operating system threads can preempt execution at arbitrary bytecode instructions, asyncio relies on cooperative multitasking where task switching occurs at explicit suspension points. However, an await expression introduces a potential suspension point rather than an unconditional context switch; awaiting an already completed future or a lock that is currently free can complete synchronously on the active frame. Concurrency bugs arise whenever a workflow evaluates shared state, encounters an await expression that yields control back to the event loop, and then modifies that state based on the earlier evaluation. Between the observation of the condition and the resulting write, any other runnable task scheduled on the event loop can execute, inspect the unchanged memory, and proceed under identical, obsolete assumptions.
The deterministic insertion of an awaited zero-second sleep inside the reproducer isolates this interleaving vulnerability. In application code, such yield points rarely appear as bare sleep statements. Instead, an asynchronous routine might perform an awaited lookup, emit an asynchronous telemetry event, query a secondary resource, or await an internal buffer flush immediately after verifying that a task is pending. Whenever an awaited operation yields execution, the event loop is free to advance any other runnable task currently queued. If another worker task is waiting to inspect that same collection, it will evaluate the shared dictionary before the original task returns to apply the mutation, creating a classic check-then-act race within a single operating system thread.
Once duplicate worker claims occur, downstream execution invariants fail rapidly within the process. When two workers both return job identifier 1, each calling frame believes it has attained exclusive ownership over that distinct payload. If these tasks proceed to perform non-idempotent operations, such as triggering an external network mutation or transforming in-memory buffers, the work is executed twice concurrently. Furthermore, the in-memory dictionary itself finishes in the running state, masking the concurrency failure from passive inspection. An inspection of the dictionary reveals a single valid status update, even though two independent task stacks received the identical identifier and are actively executing duplicate work.
Understanding this failure pattern requires keeping the concurrency model within its verified boundary. The duplicate claim reproduced here is entirely confined to tasks sharing state on a single asyncio event loop inside one process. It does not represent cross-process synchronization, database transaction isolation, or operating system thread synchronization. As Python documentation notes, asyncio synchronization primitives are designed strictly for task coordination and are explicitly not thread-safe. Preventing duplicate claims at this local level demands that the evaluation of a pending record and the subsequent mutation to the running state occur as an indivisible step without yielding control to the event loop.
Run the smallest duplicate-claim example
Python documents asyncio.Lock as a mutex for asyncio tasks and states that these synchronization primitives are not thread-safe. Python3.12 asyncio synchronization primitives
To isolate how duplicate claims manifest in memory without the noise of flaky timing or external dependencies, the reproducer strips the coordination problem down to its smallest functional components. The shared state consists of a Python list holding a single dictionary with an identifier of 1 and an initial state of pending. The broken claim function iterates over this list, inspects the state string of each record, pauses execution using an awaited zero-second sleep, and then updates the state to running before returning the integer identifier. In the verification test, two concurrent workers invoke this broken function simultaneously via asyncio gather, operating directly on the exact same list instance within a single event loop.
For reliable local publishing, coordinate in-memory work separately from remote creation. A local code check does not establish that a timed-out upload can safely be repeated.
Tracing the task scheduling reveals precisely where the mutual exclusion contract breaks down. When the event loop begins executing the first worker, that task enters the loop over the jobs list and inspects the dictionary. The condition checking whether the state equals pending evaluates to true. Immediately after this check, the coroutine executes await asyncio sleep zero. Under the asyncio cooperative multitasking model, awaiting a sleep of zero does not introduce a wall-clock delay; instead, it relinquishes execution back to the loop scheduler and places the calling task at the end of the ready queue. The critical defect is that worker one has yielded control after reading the pending status but before updating the record.
While worker one sits in the ready queue, the event loop immediately picks up the next runnable task, which is worker two entering the same broken claim routine against the same shared list. Worker two inspects the dictionary for job 1. Because worker one was paused before mutating the dictionary, worker two reads the exact same state value of pending. Worker two evaluates the branch condition as true and promptly reaches its own await asyncio sleep zero statement. Worker two yields execution back to the loop scheduler. At this exact junction, both tasks have completed their pre-claim qualification, both tasks hold a reference to job 1, and both tasks are queued to resume directly on the mutation statement.
When the scheduler cycles back to worker one, execution resumes immediately after the sleep statement. Worker one sets the state key to running and returns the job identifier 1. Worker one has now concluded its claim phase successfully. Next, the scheduler activates worker two, which resumes from its own yield point. Worker two executes the assignment, setting the state key to running a second time, and returns the job identifier 1. When asyncio gather completes, the aggregated return value is the list containing 1 and 1. Two separate task contexts have claimed the single pending unit of work, returning identical identifiers while leaving behind an in-memory dictionary that superficially indicates a normal, single transition to running.
Retaining this synthetic zero-second sleep reproducer provides a valuable regression fixture in continuous integration. Because it removes network latency and jitter, it converts an intermittent concurrency defect into a repeatable demonstration of the specific broken implementation alongside a check of the corrected implementation. Engineering teams can preserve broken_claim alongside claim_one to demonstrate to contributors why state transitions must not interleave with loop yields. As a next step, teams can add proposed regression checks—such as checking behavior when tasks are cancelled during acquisition or verifying behavior when jobs are dynamically appended during claims—noting that these proposed checks remain unperformed in the current verified baseline.
async_claim.py
import asyncio
async def broken_claim(jobs):
for job in jobs:
if job['state'] == 'pending':
await asyncio.sleep(0)
job['state'] = 'running'
return job['id']
return None
async def claim_one(jobs, lock):
async with lock:
for job in jobs:
if job['state'] == 'pending':
job['state'] = 'running'
return job['id']
return None
Recorded local check: Four local asyncio checks, including a deterministic duplicate-claim reproducer and a shared-lock correction within one event loop. No network call, cross-process coordination, persistent queue, cancellation recovery or exactly-once side-effect guarantee was tested.; environment: Python 3.12.3 on Windows
Keep the protected transition small
The correction implemented in claim_one protects the scan and mutation inside an async with lock context manager using a shared asyncio.Lock. In the verified implementation, claim_one inspects the jobs collection and commits the state transition to running without containing any await expressions inside the critical section. Because execution on a single event loop runs synchronously until an explicit suspension point is reached, the straight-line synchronous check-and-set sequence itself prevents task interleaving during the state transition. The shared lock establishes an explicit coordination protocol across all current and future cooperating workers, ensuring that any caller entering claim_one respects the mutual exclusion boundary even if surrounding asynchronous routines change over time.
In this particular implementation the first worker can acquire, scan, update and release the lock in one uninterrupted event-loop turn. The second worker therefore need not suspend while acquiring it: it can run afterward, acquire the now-free lock, observe running and return None. If another cooperating operation ever holds the same lock across a suspension, acquisition may instead wait. The four tests establish the returned claim values; they do not establish that a waiter was suspended on the lock during the corrected function. This distinction matters when explaining why a fix works: the non-yielding state transition removes the demonstrated race, while the shared lock records the coordination protocol callers must follow.
A central architectural principle demonstrated by claim_one is keeping the protected transition as small as possible. The purpose of the lock is solely to manage the transition of state, not to govern the execution of the claimed job. If a worker were to acquire the lock, mutate the job state, and then await an external payload or long-running input-output operation while still holding the lock, all other workers would be blocked from claiming any remaining items. Holding a synchronization primitive across long-running or unbounded asynchronous operations serializes the entire worker fleet, converting concurrent execution into sequential processing. The lock must be released the instant the state transition is committed so that subsequent workers can immediately access the queue and claim subsequent pending tasks.
A fundamental characteristic of claim_one is that its critical section contains zero await statements. The loop, equality comparison, and dictionary mutation execute as straight-line synchronous bytecode. On a single asyncio event loop, purely synchronous operations run to completion without interruption from other coroutines. The lock does not serve to pause or preempt synchronous execution; rather, it acts as a gate at the function entrance. By keeping the critical section entirely synchronous and minimal, the design eliminates opportunities for cooperative yielding while holding the lock, ensuring that the critical section completes in the immediate synchronous turn.
This design enforces a clean separation of concerns: the claim phase is a rapid, non-yielding state transition, whereas the processing phase is an independent asynchronous workflow executed entirely outside the lock. Once claim_one returns an identifier, the calling coroutine owns that work item locally and can proceed with whatever asynchronous operations are required without impeding other workers. However, this safety depends entirely on consistent usage across all consumer tasks. If even one worker inspects or modifies the jobs list without acquiring the shared lock, the synchronization invariant breaks down. Furthermore, because asyncio.Lock is strictly scoped to the cooperative scheduling of a single Python event loop, this protection applies only to tasks running within the same process memory space.
Read the four checks and their boundary
The verification suite for this pattern consists of four automated unit tests executed under Python 3.12.3 to validate the behavior of both the vulnerable and protected claim implementations. The first check establishes the baseline failure mode using the broken claim function. In this test, two asynchronous tasks are scheduled concurrently against a shared list containing a single pending job with an identifier of 1. Because the vulnerable function yields control to the event loop via an explicit zero-second sleep between evaluating the pending condition and updating the state dictionary, both tasks complete the condition check before either modifies the record. The test confirms that both tasks claim the same job, returning the duplicate array [1, 1]. This test verifies that cooperative scheduling alone does not prevent race conditions when state evaluation and state mutation are separated by a yield point.
The second check evaluates the single-job resolution using the corrected claim function governed by an asyncio lock. When two concurrent workers compete for the single pending job, the lock guarantees that only one task enters the critical section at a time. The first worker acquires the lock, finds the pending job, synchronously mutates its state to running, and releases the lock upon returning the identifier. When the second worker acquires the lock, the job is already marked as running, causing the function to complete the loop without finding eligible work and return None. The resulting claim list contains exactly one integer identifier 1 and one None value, proving that the lock eliminates the duplicate claim under the same scheduling conditions.
The remaining two automated tests in the verified suite evaluate boundary cases defined in test_example.py: test_empty_queue and test_many_waiters. In test_empty_queue, a worker coroutine invokes claim_one against an empty list with an asyncio.Lock instance, confirming that the function immediately returns None without raising exceptions or deadlocking. In test_many_waiters, eight concurrent worker coroutines are executed via asyncio.gather against a shared list of four pending jobs with identifiers from 0 through 3. The test confirms that all four job identifiers are claimed exactly once, that the remaining four workers return None, and that the sorted list of successful claim values matches [0, 1, 2, 3]. These tests verify claim distribution across the eight scheduled consumers within the local test scope.
Understanding the strict boundary of these tests is critical for system architecture. The unit tests verify only that an in-memory dictionary mutation is serialized across cooperative coroutines within a single event loop. The test suite did not execute any network calls, database queries, disk operations, or cross-process communications. Furthermore, the test suite did not evaluate worker task cancellation, unhandled exceptions during downstream execution, or persistent state recovery. If a worker coroutine acquires a job identifier and subsequently encounters a runtime error or cancellation during its work phase, the job remains in the running state forever because no release or timeout mechanism exists in this code.
Similarly, this verification does not support any claim of distributed exactly-once execution or cross-thread safety. Python standard library documentation explicitly notes that asyncio synchronization primitives, including asyncio.Lock, are not thread-safe. If multiple operating system threads or separate containerized processes attempt to interact with the same underlying data source using independent asyncio locks, the mutual exclusion invariant is entirely lost. The four passing checks establish correctness only for a single event loop running cooperative tasks against local objects. Any extension into distributed workers, durable storage, or background worker pools requires distinct synchronization mechanisms designed specifically for those boundaries.
A failed worker needs a separate recovery contract
When a coroutine successfully claims a job using the protected function, the lock is released immediately after updating the status to running. At this point, the worker holds the job identifier and enters its processing phase. However, mutual exclusion during the claim phase provides no guarantees about task completion. If the worker subsequently encounters an unhandled exception, a frozen socket, or an asynchronous cancellation while executing its workload, the job object remains in the running state indefinitely. Because the claim selection loop only inspects items marked as pending, an abandoned job becomes permanently inaccessible to all remaining workers, stranding the work item without ever recording a completion or failure.
Resolving this vulnerability requires an explicit recovery contract because claiming and executing belong to different operational lifecycle phases. In a single-process asyncio environment, the most immediate pattern wraps downstream execution in an exception-handling block outside the claim routine. If the worker task catches a runtime error or an asyncio.CancelledError during execution, the cleanup block must safely transition the job state back to pending or advance it to an explicit failed state. Without this targeted handling, the cooperative runtime will simply discard the cancelled or faulted task, leaving in-memory dictionary flags untouched and rendering the overall queue inconsistent over time.
Relying solely on local try-finally blocks within worker tasks introduces secondary failure modes. If a worker hangs inside an unyielding synchronous operation or an external call without proper timeouts, the task neither fails nor raises a cancellation error. The job remains locked in the running state because the worker never reaches its finally block. Addressing stalled workers typically requires an active supervisor task or an explicit lease pattern. In a lease model, the claim record stores a timestamp alongside the running status. Workers must periodically renew this timestamp while processing, or a separate cleanup coroutine will inspect running jobs, evaluate elapsed time against an allowable deadline, and reassign expired jobs back to pending.
Introducing lease expiration contracts creates a distinct engineering tradeoff: the risk of duplicate downstream execution. If a supervisor task declares a lease expired because a worker was temporarily descheduled, resource-constrained, or experiencing network latency, the supervisor resets the job to pending. A second worker can then immediately claim the same job identifier. If the original worker resumes execution, both workers will process the exact same payload simultaneously. Preventing duplicate side effects in this scenario cannot be achieved by in-memory asyncio locks; it requires external idempotency keys, version checks, or conditional writes at the persistence boundary to reject delayed commits from superseded workers.
None of these recovery mechanisms were present in the verified test suite, which strictly tested the pending-to-running transition using claim_one on a single event loop. The verified code verifies that two workers cannot simultaneously obtain the same job ID during the selection step, but it does not evaluate heartbeat loops, retry limits, dead-letter routing, or cancellation cleanup. Systems operating in production must decouple the atomic claim primitive from the subsequent failure handling policy, ensuring that the recovery contract accounts for task crashes, unhandled cancellations, and lease expirations without silently corrupting queue state.
Decide when an in-memory queue has become too small
Deciding whether to keep job coordination in-memory requires evaluating process boundaries and restart tolerance. An in-memory queue managed by an asyncio lock is appropriate when worker tasks reside entirely within a single event loop, workloads are primarily asynchronous I/O, and dropping queued tasks during an unexpected process crash is acceptable. Because tasks share heap memory, coordination requires no external infrastructure. However, if the process restarts or terminates, all uncompleted in-memory jobs are lost. Applications must determine whether tasks can simply be re-ingested or regenerated upon reboot, or whether loss of job state represents unacceptable data loss.
Managing an in-memory queue also requires implementing bounded admission control outside the claim routine. In the verified code, jobs are stored as dictionaries inside an unbounded Python list. In a production service, incoming work must be constrained before being appended to memory, using primitives like asyncio.Queue with a defined maxsize or an explicit admission semaphore to reject or back off submissions when queue depth exceeds capacity. Furthermore, job identifiers should reference separate immutable payloads rather than embedding large mutable buffers directly in the queue list, keeping memory footprints manageable and reducing cross-task mutation side effects.
A critical boundary of this architecture is process isolation. An asyncio.Lock functions strictly within a single Python event loop and cannot synchronize state across separate operating system processes. If an application scales out horizontally by running multiple process workers under a process supervisor or container orchestrator, each process operates in its own memory space with its own independent lock instance. Tasks across different processes running this in-memory code would not coordinate, resulting in each process claiming jobs independently from its own local list. When workloads scale beyond a single event loop, state coordination must shift to an external coordinator.
When transitioning from in-memory coordination to persistent storage, engineers should apply an incremental migration checklist. First, verify whether job state requires survivability across container restarts. Second, introduce durable claim mechanisms—such as an atomic SQL UPDATE with a WHERE state = 'pending' clause or a Redis atomic script—to replace the in-memory dictionary mutation. Third, separate the durable claim transaction from the downstream execution of side effects, ensuring that failures during external API calls do not leave jobs stranded in an unrecoverable state. Any extended verification of these persistent transitions represents proposed testing that must be validated in an isolated test environment.
Adopting external brokers or database-backed queues introduces significant operational overhead, including connection pooling, network timeouts, serialization costs, and reaper routines for expired leases. For lightweight internal tasks, temporary batch aggregation, or asynchronous background logging where job drops are tolerable, retaining claim logic in-memory with a shared asyncio.Lock remains a clean, zero-dependency pattern. The decision to migrate should be driven by concrete operational requirements—such as multi-process worker scaling, persistent audit logs, and crash recovery—rather than an assumption that in-memory cooperative coordination is inadequate for single-process tasks.
Prepared with AI assistance and automated editorial and meaning reviews. The supplied code was checked locally on synthetic fixtures; no production workload, customer outcome or live provider stream was tested.



