On this article, you’ll learn to construct a strong JSON serialization pipeline in Python that handles customized knowledge sorts, sleek error restoration, and schema migrations.
Matters we’ll cowl embrace:
- The best way to outline a customized JSON encoder to deal with Python sorts like
datetime,set,Path, andEnum. - The best way to write a customized decoder perform that reconstructs these authentic Python objects from a serialized JSON string, together with sleek dealing with of malformed knowledge.
- The best way to handle schema modifications over time utilizing a manufacturing facility perform that applies defaults and migrations throughout deserialization.

Introduction and Motivation
Anybody pretty skilled in developing machine studying pipelines might need seen at some stage that solely a small proportion of their time is devoted to writing the mannequin structure. What do they spend the remainder of that point on? Information plumbing: the method of cleansing, preprocessing, and reworking knowledge to make it look completely digestible for downstream fashions.
This text tackles a hidden payoff usually found alongside the machine studying engineering journey: the fixed want to maneuver structured datasets between information, processes, microservices, and APIs. All through these processes, you want an answer to serialize and deserialize knowledge in a dependable and sturdy style.
We are going to discover some sensible methods for serializing customized sorts in a means that allows parsing them again securely, in addition to managing knowledge schema modifications and coping with malformed knowledge — all with out essentially crashing the entire pipeline!
Earlier than going deep into the three-step resolution, let’s have a look at the basis drawback: JSON limitations when encountering sure knowledge sorts like datetime, set, Path, or Enum. Suppose that is the configuration for a machine studying experiment:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
from datetime import datetime from enum import Enum from pathlib import Path import json
class ModelType(Enum): RANDOM_FOREST = “rf” NEURAL_NET = “nn”
experiment_config = { “experiment_id”: 1042, “model_type”: ModelType.NEURAL_NET, “options”: {“age”, “revenue”, “location”}, # A set! “data_path”: Path(“/knowledge/v2/practice.csv”), “timestamp”: datetime.now(), “nested_meta”: { “retries”: 3 } } |
The second you attempt to serialize this experiment configuration utilizing json.dumps(experiment_config), you’ll instantly come throughout an error like this:
|
TypeError: Object of kind ModelType is not JSON serializable |
These sorts of errors are what we’re about to repair by defining a strong pipeline for encoding and decoding JSON objects with customized sorts.
1. Defining a Customized Encoder
First, we’ll outline a customized encoder class that inherits from json.JSONEncoder. That is essential to have the ability to “customise” the prevailing json.dumps() with the additional logic we’ll implement in our class to deal with customized knowledge sorts that JSON can’t deal with by default.
Moreover subclassing json.JSONEncoder, we additionally must override the default() methodology. That is the place, one after the other, we’ll outline the customized strategy to deal with every particular knowledge kind we’d must work with, utilizing conditionals within the type if isinstance(obj, DATA_TYPE):, changing DATA_TYPE with a selected knowledge kind or class title, as proven under:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
class RobustEncoder(json.JSONEncoder): def default(self, obj): # Dealing with Datetimes if isinstance(obj, datetime): return {“__type__”: “datetime”, “worth”: obj.isoformat()}
# Dealing with Units (convert to listing for JSON) if isinstance(obj, set): return {“__type__”: “set”, “worth”: listing(obj)}
# Dealing with Paths (convert to string) if isinstance(obj, Path): return {“__type__”: “path”, “worth”: str(obj)}
# Dealing with Enums (extract the underlying worth) if isinstance(obj, Enum): return obj.worth
# Letting the bottom class deal with every thing else (or increase TypeError) return tremendous().default(obj)
# Serializing our config object json_string = json.dumps(experiment_config, cls=RobustEncoder, indent=2) print(json_string) |
Yet another essential element: the additional trick for sturdy serialization within the above code is attaching the __type__ marker to things which may lose their authentic typing when they’re become JSON primitives. This happens, as an illustration, when changing Python units into lists.
Discover how we use cls=RobustEncoder on this new name to json.dumps() to make use of our customized encoder.
Consequence:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
{ “experiment_id”: 1042, “model_type”: “nn”, “options”: { “__type__”: “set”, “worth”: [ “income”, “location”, “age” ] }, “data_path”: { “__type__”: “path”, “worth”: “/knowledge/v2/practice.csv” }, “timestamp”: { “__type__”: “datetime”, “worth”: “2026-09-28T14:26:10.328310” }, “nested_meta”: { “retries”: 3 } } |
No TypeError anymore!
2. Defining a Customized Decoder
For the inverse course of — acquiring Python objects again from a serialized JSON string — the hot button is utilizing a unique methodology and argument: json.hundreds() alongside an object_hook parameter.
We don’t must outline a category by inheritance this time, however we do create a customized perform named robust_decoder which, when handed to json.hundreds(), will run on all nested dictionaries within the JSON string throughout parsing. The code contains the logic to reconstruct authentic Python objects, akin to Path, as wanted:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
import logging
def robust_decoder(dct): # Checking if this dictionary represents considered one of our customized sorts if “__type__” in dct and “worth” in dct: type_name = dct[“__type__”] val = dct[“value”]
strive: if type_name == “datetime”: return datetime.fromisoformat(val) elif type_name == “set”: return set(val) elif type_name == “path”: return Path(val) besides (ValueError, TypeError) as e: # Dealing with Malformed Information: Logging the error with out inflicting a crash. # Return a protected fallback or a specialised error dictionary. logging.warning(f“Malformed {type_name} knowledge detected: {e}. Falling again to uncooked worth.”) return val
return dct
# Deserialize again to Python objects parsed_config = json.hundreds(json_string, object_hook=robust_decoder) print(kind(parsed_config[“timestamp”])) print(kind(parsed_config[“features”])) |
The perform additionally contains exception dealing with in case malformed knowledge is encountered.
Consequence (output of the 2 print calls):
|
<class ‘datetime.datetime’> <class ‘set’> |
3. Coping with Schema Adjustments
One additional widespread problem in machine studying tasks is that mannequin metadata schemas (mannequin specs or configuration, in easier phrases) change continuously. It’s not uncommon so as to add new hyperparameters, deprecate previous ones, or rename a metric. Loading previous JSON config objects into newer variations of your code due to this fact usually requires wrapping the deserialized dictionary in an extra validation layer.
The next manufacturing facility perform will help obtain this, aided by the sturdy decoder perform we applied earlier:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
def load_experiment_config(json_data: str) -> dict: # 1. Deserializing with our sturdy hook raw_dict = json.hundreds(json_data, object_hook=robust_decoder)
# 2. Making use of Schema Migrations / Defaults # Our older configuration did not have ‘learning_rate’, so we offer a protected default. validated_config = { “experiment_id”: raw_dict.get(“experiment_id”, “unknown”), “model_type”: raw_dict.get(“model_type”, “rf”), “options”: raw_dict.get(“options”, set()), “data_path”: raw_dict.get(“data_path”, Path(“./default_data.csv”)), “timestamp”: raw_dict.get(“timestamp”, datetime.now()), “learning_rate”: raw_dict.get(“learning_rate”, 0.001), # New schema hyperparameter “nested_meta”: raw_dict.get(“nested_meta”, {}) }
# 3. Discarding deprecated fields (e.g., if ‘old_metric’ was in raw_dict, optionally drop it right here) return validated_config |
We explicitly added the defaulting mechanism for a "learning_rate" hyperparameter that didn’t exist within the authentic schema, however you’ll be able to comply with an analogous strategy for some other new hyperparameter that needs to be current in your up to date mannequin specs.
Let’s have a look at how this enhanced technique works by passing the unique JSON string by the complete pipeline:
|
print(“Uncooked JSON being loaded:”) print(json_string) print(“n” + “=”*40 + “n”)
# Processing the uncooked, authentic JSON object by our sturdy loading and validation pipeline final_config = load_experiment_config(json_string)
print(“— Validated Python Configuration —“) for key, worth in final_config.gadgets(): # Printing the important thing, the reconstructed worth, and its underlying Python kind print(f“{key}: {worth} (Kind: {kind(worth).__name__})”) |
Outcomes:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
Uncooked JSON being loaded: { “experiment_id”: 1042, “model_type”: “nn”, “options”: { “__type__”: “set”, “worth”: [ “income”, “location”, “age” ] }, “data_path”: { “__type__”: “path”, “worth”: “/knowledge/v2/practice.csv” }, “timestamp”: { “__type__”: “datetime”, “worth”: “2026-09-28T14:26:10.328310” }, “nested_meta”: { “retries”: 3 } }
========================================
—– Validated Python Configuration —– experiment_id: 1042 (Kind: int) model_type: nn (Kind: str) options: {‘revenue’, ‘location’, ‘age’} (Kind: set) data_path: /knowledge/v2/practice.csv (Kind: PosixPath) timestamp: 2026–09–28 14:26:10.328310 (Kind: datetime) learning_rate: 0.001 (Kind: float) nested_meta: {‘retries’: 3} (Kind: dict) |
Wrapping Up
Via three sensible steps, we’ve realized the right way to take full management of Python’s instruments for serializing and deserializing JSON objects. These steps could be mixed to type a strong serialization pipeline that handles customized knowledge sorts which might in any other case increase errors. Not solely can we deal with nested JSON objects by the lessons and features we outlined, however they’re additionally designed to fail gracefully when malformed knowledge is encountered. These strategies can meaningfully sharpen your knowledge plumbing abilities for machine studying engineering.

