On this article, you’ll discover ways to construct a strong JSON serialization pipeline in Python that handles customized information sorts, sleek error restoration, and schema migrations.
Matters we’ll cowl embrace:
- Easy methods to outline a customized JSON encoder to deal with Python sorts like
datetime,set,Path, andEnum. - Easy methods to write a customized decoder perform that reconstructs these authentic Python objects from a serialized JSON string, together with sleek dealing with of malformed information.
- Easy methods to handle schema modifications over time utilizing a manufacturing unit perform that applies defaults and migrations throughout deserialization.

Introduction and Motivation
Anybody pretty skilled in developing machine studying pipelines might need seen at some stage that solely a small proportion of their time is devoted to writing the mannequin structure. What do they spend the remainder of that point on? Information plumbing: the method of cleansing, preprocessing, and remodeling information to make it look completely digestible for downstream fashions.
This text tackles a hidden payoff usually found alongside the machine studying engineering journey: the fixed want to maneuver structured datasets between information, processes, microservices, and APIs. All through these processes, you want an answer to serialize and deserialize information in a dependable and strong trend.
We’ll discover some sensible methods for serializing customized sorts in a approach that permits parsing them again securely, in addition to managing information schema modifications and coping with malformed information — all with out essentially crashing the entire pipeline!
Earlier than going deep into the three-step resolution, let’s take a look at the foundation downside: JSON limitations when encountering sure information sorts like datetime, set, Path, or Enum. Suppose that is the configuration for a machine studying experiment:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
from datetime import datetime from enum import Enum from pathlib import Path import json
class ModelType(Enum): RANDOM_FOREST = “rf” NEURAL_NET = “nn”
experiment_config = { “experiment_id”: 1042, “model_type”: ModelType.NEURAL_NET, “options”: {“age”, “earnings”, “location”}, # A set! “data_path”: Path(“/information/v2/prepare.csv”), “timestamp”: datetime.now(), “nested_meta”: { “retries”: 3 } } |
The second you attempt to serialize this experiment configuration utilizing json.dumps(experiment_config), you’ll instantly come throughout an error like this:
|
TypeError: Object of kind ModelType is not JSON serializable |
These sorts of errors are what we’re about to repair by defining a strong pipeline for encoding and decoding JSON objects with customized sorts.
1. Defining a Customized Encoder
First, we’ll outline a customized encoder class that inherits from json.JSONEncoder. That is essential to have the ability to “customise” the prevailing json.dumps() with the additional logic we’ll implement in our class to deal with customized information sorts that JSON can not deal with by default.
Apart from subclassing json.JSONEncoder, we additionally must override the default() technique. That is the place, one after the other, we’ll outline the customized method to deal with every particular information kind we would must work with, utilizing conditionals within the type if isinstance(obj, DATA_TYPE):, changing DATA_TYPE with a particular information kind or class identify, as proven beneath:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
class RobustEncoder(json.JSONEncoder): def default(self, obj): # Dealing with Datetimes if isinstance(obj, datetime): return {“__type__”: “datetime”, “worth”: obj.isoformat()}
# Dealing with Units (convert to checklist for JSON) if isinstance(obj, set): return {“__type__”: “set”, “worth”: checklist(obj)}
# Dealing with Paths (convert to string) if isinstance(obj, Path): return {“__type__”: “path”, “worth”: str(obj)}
# Dealing with Enums (extract the underlying worth) if isinstance(obj, Enum): return obj.worth
# Letting the bottom class deal with all the pieces else (or increase TypeError) return tremendous().default(obj)
# Serializing our config object json_string = json.dumps(experiment_config, cls=RobustEncoder, indent=2) print(json_string) |
Yet another essential element: the additional trick for strong serialization within the above code is attaching the __type__ marker to things that may lose their authentic typing when they’re became JSON primitives. This happens, as an example, when changing Python units into lists.
Discover how we use cls=RobustEncoder on this new name to json.dumps() to make use of our customized encoder.
Consequence:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
{ “experiment_id”: 1042, “model_type”: “nn”, “options”: { “__type__”: “set”, “worth”: [ “income”, “location”, “age” ] }, “data_path”: { “__type__”: “path”, “worth”: “/information/v2/prepare.csv” }, “timestamp”: { “__type__”: “datetime”, “worth”: “2026-09-28T14:26:10.328310” }, “nested_meta”: { “retries”: 3 } } |
No TypeError anymore!
2. Defining a Customized Decoder
For the inverse course of — acquiring Python objects again from a serialized JSON string — the bottom line is utilizing a special technique and argument: json.hundreds() alongside an object_hook parameter.
We don’t must outline a category by means of inheritance this time, however we do create a customized perform named robust_decoder which, when handed to json.hundreds(), will run on all nested dictionaries within the JSON string throughout parsing. The code contains the logic to reconstruct authentic Python objects, corresponding to Path, as wanted:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
import logging
def robust_decoder(dct): # Checking if this dictionary represents certainly one of our customized sorts if “__type__” in dct and “worth” in dct: type_name = dct[“__type__”] val = dct[“value”]
attempt: if type_name == “datetime”: return datetime.fromisoformat(val) elif type_name == “set”: return set(val) elif type_name == “path”: return Path(val) besides (ValueError, TypeError) as e: # Dealing with Malformed Information: Logging the error with out inflicting a crash. # Return a secure fallback or a specialised error dictionary. logging.warning(f“Malformed {type_name} information detected: {e}. Falling again to uncooked worth.”) return val
return dct
# Deserialize again to Python objects parsed_config = json.hundreds(json_string, object_hook=robust_decoder) print(kind(parsed_config[“timestamp”])) print(kind(parsed_config[“features”])) |
The perform additionally contains exception dealing with in case malformed information is encountered.
Consequence (output of the 2 print calls):
|
<class ‘datetime.datetime’> <class ‘set’> |
3. Coping with Schema Adjustments
One additional frequent problem in machine studying tasks is that mannequin metadata schemas (mannequin specs or configuration, in easier phrases) change continually. It isn’t uncommon so as to add new hyperparameters, deprecate previous ones, or rename a metric. Loading previous JSON config objects into newer variations of your code due to this fact usually requires wrapping the deserialized dictionary in a further validation layer.
The next manufacturing unit perform will help obtain this, aided by the strong decoder perform we carried out earlier:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
def load_experiment_config(json_data: str) -> dict: # 1. Deserializing with our strong hook raw_dict = json.hundreds(json_data, object_hook=robust_decoder)
# 2. Making use of Schema Migrations / Defaults # Our older configuration did not have ‘learning_rate’, so we offer a secure default. validated_config = { “experiment_id”: raw_dict.get(“experiment_id”, “unknown”), “model_type”: raw_dict.get(“model_type”, “rf”), “options”: raw_dict.get(“options”, set()), “data_path”: raw_dict.get(“data_path”, Path(“./default_data.csv”)), “timestamp”: raw_dict.get(“timestamp”, datetime.now()), “learning_rate”: raw_dict.get(“learning_rate”, 0.001), # New schema hyperparameter “nested_meta”: raw_dict.get(“nested_meta”, {}) }
# 3. Discarding deprecated fields (e.g., if ‘old_metric’ was in raw_dict, optionally drop it right here) return validated_config |
We explicitly added the defaulting mechanism for a "learning_rate" hyperparameter that didn’t exist within the authentic schema, however you may observe an analogous method for every other new hyperparameter that needs to be current in your up to date mannequin specs.
Let’s take a look at how this enhanced technique works by passing the unique JSON string by means of the complete pipeline:
|
print(“Uncooked JSON being loaded:”) print(json_string) print(“n” + “=”*40 + “n”)
# Processing the uncooked, authentic JSON object by means of our strong loading and validation pipeline final_config = load_experiment_config(json_string)
print(“— Validated Python Configuration —“) for key, worth in final_config.gadgets(): # Printing the important thing, the reconstructed worth, and its underlying Python kind print(f“{key}: {worth} (Sort: {kind(worth).__name__})”) |
Outcomes:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
Uncooked JSON being loaded: { “experiment_id”: 1042, “model_type”: “nn”, “options”: { “__type__”: “set”, “worth”: [ “income”, “location”, “age” ] }, “data_path”: { “__type__”: “path”, “worth”: “/information/v2/prepare.csv” }, “timestamp”: { “__type__”: “datetime”, “worth”: “2026-09-28T14:26:10.328310” }, “nested_meta”: { “retries”: 3 } }
========================================
—– Validated Python Configuration —– experiment_id: 1042 (Sort: int) model_type: nn (Sort: str) options: {‘earnings’, ‘location’, ‘age’} (Sort: set) data_path: /information/v2/prepare.csv (Sort: PosixPath) timestamp: 2026–09–28 14:26:10.328310 (Sort: datetime) learning_rate: 0.001 (Sort: float) nested_meta: {‘retries’: 3} (Sort: dict) |
Wrapping Up
Via three sensible steps, we have now realized easy methods to take full management of Python’s instruments for serializing and deserializing JSON objects. These steps may be mixed to type a strong serialization pipeline that handles customized information sorts which might in any other case increase errors. Not solely can we deal with nested JSON objects by means of the lessons and capabilities we outlined, however they’re additionally designed to fail gracefully when malformed information is encountered. These methods can meaningfully sharpen your information plumbing abilities for machine studying engineering.








