For a Python job producing a sequence of independent records, JSON Lines is often clearer than repeatedly appending JSON objects to a normal JSON document. Serialize each record without pretty-printing, then write a newline after it. Read the result one line at a time and parse each line separately. A file containing several JSON Lines records is not one ordinary JSON document.
Understand the framing problem first
The Python json documentation warns that JSON is not a framed protocol: repeatedly calling dump on the same file does not produce a valid single JSON document. Two objects written back to back lack a surrounding array and its separators. Adding arbitrary whitespace does not turn them into one document.
The JSON Lines format documentation defines a separate convention: UTF-8 text, one valid JSON value per line, and newline record separators. A trailing newline is recommended. Blank lines are not valid records. Although objects are common, arrays, strings, numbers and null can also be valid JSON Lines values. Your application’s schema can impose a stricter object-only rule.
Try a disposable round-trip example
This example uses only Python’s standard library. It creates a temporary file, writes two records, reads them back, checks that their values survived, and removes the workspace automatically. The first note contains a real newline in memory. JSON serialization escapes it inside the record, so it does not become a physical record boundary.
import json
from pathlib import Path
from tempfile import TemporaryDirectory
records = [
{"id": 1, "note": "first line\nsecond line"},
{"id": 2, "note": "café"},
]
with TemporaryDirectory() as directory:
path = Path(directory) / "events.jsonl"
with path.open("w", encoding="utf-8", newline="\n") as stream:
for record in records:
encoded = json.dumps(record, ensure_ascii=False, allow_nan=False)
stream.write(encoded + "\n")
restored = []
with path.open(encoding="utf-8") as stream:
for line_number, line in enumerate(stream, start=1):
try:
value = json.loads(line)
except json.JSONDecodeError as exc:
raise ValueError(f"Invalid JSON at line {line_number}") from exc
if not isinstance(value, dict):
raise ValueError(f"Expected an object at line {line_number}")
restored.append(value)
assert restored == records
assert len(path.read_bytes().splitlines()) == 2
assert path.read_bytes().endswith(b"\n")
print("Two records restored, including an embedded newline")
Keep the two newline roles separate. The newline inside the note is data; the newline added after encoded is framing. Do not manually replace all newlines in the serialized text or concatenate record dictionaries using string formatting. Let the JSON encoder handle quotes, backslashes and control characters.
Why pretty-printing breaks this reader
Adding indent to dumps makes an object span several physical lines. That can be useful for a normal JSON document, but this reader expects each individual line to contain a complete value. Leave indentation off when producing JSON Lines. Readable spaces between keys and values are fine; the problem is physical line breaks within the representation of one record.
Similarly, do not parse the entire file with one json.load call and expect multiple records to be returned automatically. For a consumer requiring one JSON array, explicitly build or stream that array using a suitable implementation instead. Agree on the format with the receiving service before changing a production export.
Reject bad records without hiding the location
The reader reports a physical line number and preserves the original decoding exception. A blank line reaches the parser and fails rather than silently disappearing. A valid JSON null reaches the object check and is rejected because this particular application expects dictionaries. Neither decision means null is forbidden by the JSON Lines format itself.
For unattended jobs, decide whether a bad record must stop the whole import or be quarantined with an explicit failure report. Do not catch every exception and continue silently: that can make an incomplete import look successful. Avoid logging entire sensitive records just to identify a parsing error. Line numbers and a protected diagnostic file may be enough.
What this example does not guarantee
The temporary round trip checks encoding and framing, not concurrent writer safety, durable storage, schema completeness or exactly-once delivery. A process can still leave an incomplete final record after interruption. Multiple writers need coordination; newline separators alone are not a lock. If the file represents a complete replacement snapshot, consider the separate pattern in writing automation state files without partial JSON, adapting the format deliberately rather than confusing a snapshot with an append log.
Before deployment, test an embedded newline, Unicode text, a blank line, a truncated object and a valid non-object value. Keep the consumer’s object/schema requirements separate from basic JSON syntax validation.
This tutorial is for educational purposes. Test carefully before production use. The site owner is not responsible for misuse, damage, data loss, or production issues.
