An Interrupted Manifest Write Should Not Truncate the Previous File
Reproduce a one-byte overwrite, then write a complete staging file beside the target and replace it only after the inspected preparation step.

The transition from b'old' to b'n' in test_broken_truncates demonstrates the immediate hazard of writing directly to an active file path. Code paths in this inspection operate strictly on trusted caller-owned files, without establishing any filesystem benchmark or guarantee of power-loss durability. In the reproduction setup, an existing manifest file initialized with the content b'old' was passed to broken_write alongside the intended payload b'new'. The broken_write function opens the target path using standard write-binary mode, writes only the first byte of the replacement payload via stream.write(data[:1]), and immediately raises a RuntimeError('synthetic interruption'). Inspecting the file on disk after the failure revealed that the original three bytes were completely gone, replaced by the single byte b'n'. The target was not preserved in its prior valid state, nor did it contain the complete new payload. Instead, an unhandled exception during the write phase left behind a corrupt, truncated remnant that broke both the previous and future expectations of caller applications.
Compare the old file with the one-byte interrupted output
This destructive outcome stems directly from how standard stream opening semantics operate at the operating system level. When Python issues an open call using wb mode, the underlying filesystem layer immediately applies truncation flags such as O_TRUNC to the directory entry and its allocated blocks. This truncation occurs before the application sends any data across the handle, reducing the file length to zero bytes at the very beginning of the operation. If a subsequent write fails due to a network interruption, an unhandled formatting error, an out-of-memory condition, or an explicit exception, the file descriptor closes without restoring the cleared contents. The prior data cannot be recovered from disk buffers because the metadata and block allocations were discarded upon open. For manifests, state files, and configuration records, even a single partial byte or an empty file constitutes fatal data corruption.
Decoupling the file generation phase from the active destination entry is therefore the foundational requirement for robust manifest management. In the local unit test suite, test_broken_truncates asserts that broken_write raises a RuntimeError and explicitly confirms via self.path.read_bytes() that the path contains b'n' rather than b'old'. This test verifies the failure mode in isolation: opening an existing target in place directly destroys the known-good baseline before the replacement stream can establish integrity. To prevent this truncation, the preparation of incoming data must take place entirely out of band, using a separate location that can be abandoned safely if an exception occurs during serialization, streaming, or pre-replacement inspection.
Adopting a Python staging file replace manifest workflow addresses this exact failure by holding the target intact until the new bytes are completely flushed and ready on disk. However, the scope of this behavior must remain technically precise. The verified tests recorded here demonstrate this mechanism exclusively on Windows under Python 3.12.3 using workspace-owned temporary directories. They confirm that a staging file written beside the target isolates intermediate write failures, but they do not claim power-cut resilience, hardware write-barrier guarantees, or safety against simultaneous, uncoordinated concurrent writers competing for the same directory entry.
Run the staging writer in an owned temporary directory
Operating an atomic replacement workflow requires strict control over where intermediate files are materialized. In the implementation of replace_bytes, the staging file is allocated by calling tempfile.mkstemp with prefix set to '.' + target.name + '.' and dir set to target.parent. Directing mkstemp to use target.parent is a deliberate architectural constraint dictated by how underlying filesystems handle directory operations. The subsequent call to os.replace relies on renaming an existing directory entry directly to the final destination path. If the staging file were created in the default operating system temporary directory, the target and staging paths could easily reside on separate mount points, partitions, or virtual volumes. Renaming across distinct filesystem boundaries cannot be performed as an atomic pointer switch and triggers an OSError. Confining the staging file to the target's immediate parent directory ensures that both filesystem entries share the exact same storage volume.
os.replace replaces a destination file when permitted; a same-filesystem staging design is separate from a claim about crash durability or concurrent-writer ownership. Python3.12 os.replace and fsync
The local unit test suite isolates this behavior by constructing an explicitly owned temporary directory on the local filesystem. In test_example.py, the WriteTests setUp fixture invokes tempfile.TemporaryDirectory with prefix='atomic-case-' and dir=HERE, resolving the resulting path to an absolute location stored in self.root. To verify boundary containment before running any writes, the fixture executes an assertion checking that self.root.is_relative_to(HERE). The initial manifest path is then anchored inside this owned folder as self.root / 'manifest.json' and seeded with b'old'. When teardown occurs, the clean method asserts once again that Path(self.temp.name).resolve().is_relative_to(HERE) before invoking cleanup on the temporary directory handle. This structure guarantees that every test runs against a clean directory tree where staging file allocation, renaming, and removal do not leak into adjacent project folders or unmanaged global temporary locations.
In the local publishing workflow, writing an artifact is one boundary among several. Replacing one local file does not atomically update a remote post, a journal and an uploaded image; those operations need their own receipts and reconciliation rules.
Confining writes to caller-owned paths also clarifies the operational boundaries and security assumptions of the helper. The implementation converts the incoming target argument directly using Path(target). It assumes that the caller provides a trusted, validated filesystem path. The code does not inspect paths for directory traversal sequences, resolve canonical symlink hierarchies, or enforce isolation against hostile input strings. If the parent directory does not exist or lacks appropriate write permissions, mkstemp raises an immediate FileNotFoundError or PermissionError. The helper relies on the surrounding application to govern directory permissions, establish access controls, and confirm path validity before attempting data serialization.
Finally, allocating the staging file with mkstemp avoids collisions between distinct preparation cycles. By combining dir=target.parent with a dot-prefixed target name, the temporary file helper opens a unique file descriptor using operating-system-level exclusive creation flags. Even if two independent processes attempt to prepare a replacement for the same target manifest simultaneously, each process receives a distinct, uniquely suffixed staging file beside the target. Although this mechanism does not serialize which process ultimately wins the subsequent os.replace call, it guarantees that competing callers never corrupt or overwrite each other's intermediate byte streams during the write phase.
atomic_save.py
from pathlib import Path
import os
import tempfile
def broken_write(target, data):
with open(target, 'wb') as stream:
stream.write(data[:1])
raise RuntimeError('synthetic interruption')
def replace_bytes(target, data, before_replace=None):
if not isinstance(data, bytes):
raise TypeError('Expected bytes')
target = Path(target)
descriptor, name = tempfile.mkstemp(prefix='.'+target.name+'.', dir=target.parent)
staging = Path(name)
try:
with os.fdopen(descriptor, 'wb') as stream:
stream.write(data)
stream.flush()
os.fsync(stream.fileno())
if before_replace is not None:
before_replace(staging, target)
os.replace(staging, target)
finally:
if staging.exists():
staging.unlink()
Recorded local check: Seven local filesystem tests in verified workspace-owned temporary directories on Windows. os.fsync is actually called on the staging file; failures are injected with a hook or mocked replace call. No process kill, power loss, directory fsync, cross-filesystem move, concurrent writer protection, permission/ownership preservation or production durability was tested. Paths are trusted caller inputs, not a sandbox. JSON content is not parsed. Cleanup targets only the uniquely created staging file; the tests verify their resolved workspace roots before temporary-directory cleanup.; environment: Python3.12.3 on Windows
Keep the staging handle closed before replacement
The implementation of replace_bytes establishes an explicit boundary between writing bytes to disk and modifying the directory namespace. When tempfile.mkstemp allocates a temporary file, it returns both a low-level integer file descriptor and the resolved string path. The helper wraps this integer descriptor in a buffered binary writer using os.fdopen(descriptor, 'wb'). By encapsulating the write operations within a dedicated with block, the design guarantees that the underlying file handle is closed as soon as the block exits, well before the staging path is passed to os.replace.
The with block closes the staging stream before the helper calls before_replace or os.replace. That ordering is visible in the source and was used in all seven Windows tests. The suite does not attempt to rename a still-open staging file, so it cannot establish a universal Windows sharing rule or a guaranteed exception for that alternative sequence. Its replacement-failure case patches os.replace to raise a synthetic PermissionError. That case tests cleanup and preservation of old bytes under an injected exception, independently of what would cause a real operating-system failure.
This structural separation also dictates the precise ordering of stream synchronization. To ensure all buffered bytes reach the kernel, the function calls stream.flush() followed by os.fsync(stream.fileno()). The fsync system call requires an active, valid integer file descriptor; attempting to synchronize storage after closing the stream would fail with an OSError or ValueError indicating a closed file. Therefore, all data transfer and synchronization must happen strictly within the descriptor lifecycle. Exiting the with block then closes the underlying descriptor cleanly, leaving the staging file on disk in a fully written, synchronized, and closed state.
Positioning the closure before the before_replace inspection hook further stabilizes preparation. If a caller supplies an inspection callback to verify file integrity, that hook operates on a closed staging file rather than an active write stream. An inspector can safely open the staging path for reading, verify its byte length, or calculate cryptographic digests without contending with buffered write caches or file locks. Once the inspection completes without raising an exception, os.replace performs the final metadata exchange on two completely idle paths. Closing the staging handle early transforms the file from an active I/O buffer into an inert filesystem entry ready for atomic substitution.
Trace each injected failure and its cleanup
The test suite uses two distinct failure injection points to verify that partial execution leaves both the original target intact and the directory clean of orphaned staging files. Rather than relying on non-deterministic external faults, the tests simulate deterministic exceptions within the helper execution lifecycle. The first injection point intercepts the sequence through the optional before_replace hook, while the second simulates an operating system failure during the os.replace invocation using unittest.mock.patch. Both paths exercise the try-finally structure of replace_bytes under controlled conditions on the Windows filesystem.
In test_hook_failure_preserves_old, the caller passes a function defining def fail(staging, target): raise RuntimeError('injected before replacement'). At this juncture, the staging file has already been opened, written with complete bytes, flushed to the operating system, synchronized using os.fsync, and closed. When the helper invokes the hook, the exception immediately halts execution before os.replace is ever reached. Control shifts directly to the finally block. The helper checks staging.exists() and calls staging.unlink(). The unit test asserts two conditions: first, reading the target manifest.json yields the original byte string b'old', confirming that no truncation occurred; second, enumerating the directory contents via [p.name for p in self.root.iterdir()] returns solely ['manifest.json']. The staging artifact is successfully removed, leaving no auxiliary files.
The second injected failure exercises the boundary where preparation succeeds, but the filesystem rejects the substitution itself. In test_replace_failure_cleanup, os.replace is patched with patch('atomic_save.os.replace', side_effect=PermissionError('synthetic')), raising a PermissionError that mimics Windows file access conflicts or transient permission denials. Just as with the hook error, the exception surfaces after the staging stream is closed. Because os.replace failed to execute, the original destination file remains unmodified with its prior b'old' content. Control again transfers to the finally block. The assertion verifies that len(list(self.root.iterdir())) equals one, demonstrating that staging.unlink deleted the unlinked temporary file even when the core replacement operation threw an error.
While the finally block reliably removes temporary staging files during standard Python exception unwinding, this cleanup mechanism operates under clear boundaries. The seven executed tests confirm cleanup only when Python runtime control flow completes normal unwinding through the finally clause. If the staging.unlink call itself were to fail, such as when an external process holds an exclusive lock on the newly created staging file on Windows, an unhandled PermissionError would emerge during cleanup, leaving an orphan on disk. Similarly, sudden process termination, such as an immediate interpreter crash or power cut, bypasses Python exception handling entirely, preventing finally from executing. The test suite verifies clean exception unwinding for handled runtime errors, not absolute resilience against unmanaged operating system terminations.
Separate byte replacement from JSON validation
In the verified local test suite, replace_bytes treats the supplied data strictly as an opaque sequence of raw bytes. Although the test fixture names its primary target file manifest.json, the replacement helper possesses no internal awareness of JSON syntax, encoding rules, or structural constraints. The implementation strictly enforces data typing at the entry point through if not isinstance(data, bytes): raise TypeError('Expected bytes'); test_reject_text verifies that passing a string such as 'text' raises a TypeError before any staging file or directory handle is allocated. Transforming structured in-memory records into an encoded byte sequence is the explicit duty of the caller, establishing a clean boundary between data serialization and filesystem coordination.
Coupling data validation directly to disk staging introduces unnecessary failure modes into storage routines. If a function attempts to stream an in-memory dictionary directly to a staging descriptor using a serialization tool like json.dump, any serialization defect—such as an unhandled non-serializable object, a circular reference, or an encoding error—surfaces while the temporary file is open on disk. Although the finally block in replace_bytes cleanly unlinks the staging file during ordinary Python exception unwinding, interleaving data serialization with low-level file descriptor management muddies error boundaries. Generating and validating complete bytes in memory before touching the disk ensures that staging resources are only created when the entire payload is already verified and ready to persist.
The replace_bytes interface does provide an optional before_replace hook that runs after the staging file is flushed, synchronized, and closed, but before os.replace is called. A caller could theoretically use this hook to re-read the staging path from disk and verify its contents against json.loads or a schema validator before the target file is overwritten. However, post-write inspection introduces practical operational costs, including redundant disk reads and additional surface area for filesystem access conflicts. Validating that a document parses as well-formed JSON after it has already been formatted, written, and synced to disk is fundamentally late. Structural validation, required field checks, and semantic integrity checks are far more effectively performed in caller memory before calling the staging helper.
The verified test execution recorded no interactions with the Python json module; it tested only literal byte sequences such as b'new', b'old', and b''. Preserving this agnostic design allows replace_bytes to serve as a reusable byte-level primitive. A higher-level manifest manager should serialize its internal state to UTF-8 encoded bytes, assert semantic correctness against domain rules, and pass the finalized byte buffer to replace_bytes. Separating byte replacement from content validation ensures that serialization bugs, schema violations, and filesystem errors fail independently, preventing malformed data constructs from ever initiating a physical write sequence.
Choose ownership rules for competing writers
A single replacement step does not coordinate multiple writers. Consider a proposed application in which A and B both read the same initial manifest, compute different updates and each call replace_bytes. This helper contains no lock or version comparison connecting either read to its later replacement. The example therefore provides no lost-update protection. That is a consequence of the helper interface; no concurrent writer experiment was run, and no cross-filesystem atomicity or crash-durability claim follows from the local tests.
Addressing concurrent modification scenarios requires external synchronization or strict ownership rules that govern who is permitted to write to the target path. In many single-instance service architectures, restricting manifest updates to a designated worker process or utilizing an advisory locking mechanism eliminates the possibility of overlapping write cycles. When advisory locks cannot be established or when multiple independent administrative tools need modification rights, the staging file naming convention itself can incorporate process-specific identifiers such as process IDs or unique UUID components. While unique naming prevents staging files from colliding with one another prior to replacement, it does not solve the underlying race condition where the final os.replace call overwrites concurrent changes without warning.
A separate version or modification-time check followed by os.replace would still leave a gap in which another writer could change the target. Adding a version field to the bytes does not close that gap. A proposed design would need coordination across the complete read, decision and replacement sequence, or a storage operation that performs a conditional update transactionally. Those designs require their own implementation and competing-writer tests. The current helper should be used under an explicit single-writer assumption rather than described as an optimistic compare-and-swap.
The verified test execution environment for replace_bytes operated under a single-threaded test harness where test cases possessed exclusive access to their temporary directories. No concurrent writer protection mechanisms, multi-process synchronization primitives, or distributed locking protocols were executed or validated during the recorded test runs. Consequently, any deployment of the byte-replacement helper in a multi-writer environment must implement explicit queueing, database-backed locking, or directory-level coordination outside of the core file replacement function. Assuming that atomic file replacement automatically resolves concurrent write conflicts represents a frequent operational misunderstanding that can lead to subtle data loss in production environments.
Record what a successful fsync call does not establish
Invoking the os.fsync system call on an open file descriptor forces the operating system to flush any buffered data and associated metadata from volatile memory buffers down to the underlying persistent storage device. Within the implementation of the replacement helper, os.fsync(stream.fileno()) is explicitly executed against the staging file handle immediately after writing the complete payload and before closing the stream or initiating the final file replacement step. While this guarantees that the bytes written to the staging file are safely persisted on the storage medium prior to the rename operation, understanding the exact boundaries of what fsync achieves is critical for accurate system design and avoids overestimating durability guarantees.
A frequent conceptual misunderstanding involves assuming that calling fsync on a staging file automatically ensures that the parent directory entry itself is durable and correctly ordered on disk. In many operating systems and file system architectures, renaming a file via os.replace modifies the directory structure to point to the new inode, but the directory changes are buffered separately from the file contents. The verified local test runs in the workspace environment confirmed that fsync successfully flushes the target staging file descriptor, but the test scope did not exercise directory-level fsync operations or measure recovery behaviors under simulated power failure conditions. Local replacement does not extend to power-loss durability or guaranteed atomicity on every filesystem. Relying solely on file-level synchronization without directory-level synchronization means that a sudden power loss immediately after os.replace could theoretically leave the directory entry in an inconsistent state on certain file systems, even though the staging file payload was fully synchronized.
Furthermore, a successful fsync call provides no guarantees regarding higher-level application semantics, such as whether the written byte stream constitutes valid application data or syntactically correct markup. The byte-replacement helper operates strictly at the binary level, accepting arbitrary bytes and ensuring that those exact bytes are transferred from memory to the staging file without truncation. It performs no internal parsing, validation, or structural inspection of the data being written. Consequently, if an application passes malformed JSON, truncated strings, or corrupted configuration parameters to the writer, the fsync function will successfully persist those invalid bytes to the staging file, and os.replace will subsequently overwrite the target manifest with the corrupted payload.
To bridge the gap between low-level persistence and application-level correctness, proposed integration patterns often incorporate validation steps prior to invoking the file replacement helper. In a proposed production architecture, a worker process might decode the serialized payload into an in-memory data structure, verify all required schema fields, and re-serialize the validated data before passing the resulting byte string to the staging writer. However, because this validation logic was not part of the recorded test scope, developers must evaluate whether their specific deployment requires such pre-flight checks or if external monitoring tools should handle structural validation. Keeping the byte-replacement helper narrowly focused on file staging and atomic replacement maintains a clean separation of concerns, ensuring that the function remains predictable, testable, and free from unwarranted assumptions about the data it carries.
Integrate the helper without inventing a durable transaction
Adopting a staging-file replacement pattern within an existing application requires careful attention to boundary definitions and error handling expectations. Developers often look at file-system primitives like os.replace and assume they provide a complete database-style transaction across the entire storage layer. In practice, the recorded local execution scope demonstrates only that a single staging file is written, synchronized, and moved into place within a specific workspace directory on Windows. Translating this narrow mechanism into a broader application context means accepting that no underlying guarantees are established beyond what the operating system provides for a single file rename operation.
When integrating the replacement helper into an application workflow, callers must supply trusted paths and manage their own exception flows. The function itself does not wrap database connections, coordinate network shares, or handle distributed locking mechanisms. If multiple threads or separate processes attempt to write to the same manifest path simultaneously, the last process to execute os.replace will overwrite the target file without regard to what intermediary changes another writer might have introduced. Therefore, integration strategies must account for external concurrency through separate synchronization primitives or process-level locking if multiple actors share the same target file.
Furthermore, adopting this utility does not eliminate the need for application-level cleanup or state management when unexpected exceptions occur outside the file writer. While the helper guarantees that its own uniquely generated staging file is unlinked in a finally block during normal execution and specific injected failures, broader application routines must decide how to log or report failures when the replacement step itself raises an exception like PermissionError. Treating the file writer as a standalone utility rather than an all-encompassing transactional framework prevents misplaced reliance on filesystem behaviors that are outside the scope of the tested Python implementation.
Architectural planning documents should explicitly distinguish between actual observed test behaviors and proposed integration patterns that have not been subjected to empirical verification. For example, proposing that the staging writer will automatically recover from power failure, hardware degradation, or abrupt operating system termination goes beyond the documented test scope of seven local filesystem tests. Engineers designing resilient configuration managers or state persistence layers must evaluate their specific storage hardware and operating system guarantees independently, ensuring that the limits of the staging-file approach are clearly understood across the engineering team.
Prepared with AI assistance and automated reviews. The original code was exercised locally within the recorded fixture scope. No human field test, image-provider call or generated-video experiment is claimed.



