How can I json-serialize a custom iterable?











up vote
4
down vote

favorite
1












I'd like to create a type that behaves as a named tuple except that it has a custom representation, which is also respected when serialized as JSON.



The naive by-the-books approach would be something like this:



from typing import NamedTuple
import json


class MyPair(NamedTuple):
left: str
right: str

def __repr__(self):
return self.left + ':' + self.right


class MyJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, MyPair):
return str(obj)
return json.JSONEncoder.default(self, obj)


Now print(MyPair('a', 'b')) will output a:b as intended, but print(json.dumps([MyPair('a', 'b')], cls=MyJSONEncoder)) will produce [["a", "b"]] because default() is only called if an object is not primitively serializable as JSON. Since my own type is a tuple, it will be serialized before I get a chance to intervene.



Is there any nice or not-so-nice way of achieving this without making MyPair not a Tuple or iterating over the entire document in a preprocessing step that replaces all MyPair objects by strings?



Edit: To address Joran's answer, I still want to retain the ability to serialize complex trees that just contain the occasional MyPairs. My minimal example might not have made that clear, sorry.










share|improve this question
























  • Looks like you'd need to subclass JSONEncoder and override its encode and _make_iterencode methods to special-case instances of your class.
    – snakecharmerb
    Nov 10 at 11:19












  • @snakecharmerb Yes, I had a look at those methods but they don't seem to be designed to be extended so I'd end up duplicating most of the functionality of JSONEncoder :/
    – Christian
    Nov 12 at 14:37















up vote
4
down vote

favorite
1












I'd like to create a type that behaves as a named tuple except that it has a custom representation, which is also respected when serialized as JSON.



The naive by-the-books approach would be something like this:



from typing import NamedTuple
import json


class MyPair(NamedTuple):
left: str
right: str

def __repr__(self):
return self.left + ':' + self.right


class MyJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, MyPair):
return str(obj)
return json.JSONEncoder.default(self, obj)


Now print(MyPair('a', 'b')) will output a:b as intended, but print(json.dumps([MyPair('a', 'b')], cls=MyJSONEncoder)) will produce [["a", "b"]] because default() is only called if an object is not primitively serializable as JSON. Since my own type is a tuple, it will be serialized before I get a chance to intervene.



Is there any nice or not-so-nice way of achieving this without making MyPair not a Tuple or iterating over the entire document in a preprocessing step that replaces all MyPair objects by strings?



Edit: To address Joran's answer, I still want to retain the ability to serialize complex trees that just contain the occasional MyPairs. My minimal example might not have made that clear, sorry.










share|improve this question
























  • Looks like you'd need to subclass JSONEncoder and override its encode and _make_iterencode methods to special-case instances of your class.
    – snakecharmerb
    Nov 10 at 11:19












  • @snakecharmerb Yes, I had a look at those methods but they don't seem to be designed to be extended so I'd end up duplicating most of the functionality of JSONEncoder :/
    – Christian
    Nov 12 at 14:37













up vote
4
down vote

favorite
1









up vote
4
down vote

favorite
1






1





I'd like to create a type that behaves as a named tuple except that it has a custom representation, which is also respected when serialized as JSON.



The naive by-the-books approach would be something like this:



from typing import NamedTuple
import json


class MyPair(NamedTuple):
left: str
right: str

def __repr__(self):
return self.left + ':' + self.right


class MyJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, MyPair):
return str(obj)
return json.JSONEncoder.default(self, obj)


Now print(MyPair('a', 'b')) will output a:b as intended, but print(json.dumps([MyPair('a', 'b')], cls=MyJSONEncoder)) will produce [["a", "b"]] because default() is only called if an object is not primitively serializable as JSON. Since my own type is a tuple, it will be serialized before I get a chance to intervene.



Is there any nice or not-so-nice way of achieving this without making MyPair not a Tuple or iterating over the entire document in a preprocessing step that replaces all MyPair objects by strings?



Edit: To address Joran's answer, I still want to retain the ability to serialize complex trees that just contain the occasional MyPairs. My minimal example might not have made that clear, sorry.










share|improve this question















I'd like to create a type that behaves as a named tuple except that it has a custom representation, which is also respected when serialized as JSON.



The naive by-the-books approach would be something like this:



from typing import NamedTuple
import json


class MyPair(NamedTuple):
left: str
right: str

def __repr__(self):
return self.left + ':' + self.right


class MyJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, MyPair):
return str(obj)
return json.JSONEncoder.default(self, obj)


Now print(MyPair('a', 'b')) will output a:b as intended, but print(json.dumps([MyPair('a', 'b')], cls=MyJSONEncoder)) will produce [["a", "b"]] because default() is only called if an object is not primitively serializable as JSON. Since my own type is a tuple, it will be serialized before I get a chance to intervene.



Is there any nice or not-so-nice way of achieving this without making MyPair not a Tuple or iterating over the entire document in a preprocessing step that replaces all MyPair objects by strings?



Edit: To address Joran's answer, I still want to retain the ability to serialize complex trees that just contain the occasional MyPairs. My minimal example might not have made that clear, sorry.







python json python-3.x types






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 8 at 11:17

























asked Nov 7 at 16:45









Christian

270218




270218












  • Looks like you'd need to subclass JSONEncoder and override its encode and _make_iterencode methods to special-case instances of your class.
    – snakecharmerb
    Nov 10 at 11:19












  • @snakecharmerb Yes, I had a look at those methods but they don't seem to be designed to be extended so I'd end up duplicating most of the functionality of JSONEncoder :/
    – Christian
    Nov 12 at 14:37


















  • Looks like you'd need to subclass JSONEncoder and override its encode and _make_iterencode methods to special-case instances of your class.
    – snakecharmerb
    Nov 10 at 11:19












  • @snakecharmerb Yes, I had a look at those methods but they don't seem to be designed to be extended so I'd end up duplicating most of the functionality of JSONEncoder :/
    – Christian
    Nov 12 at 14:37
















Looks like you'd need to subclass JSONEncoder and override its encode and _make_iterencode methods to special-case instances of your class.
– snakecharmerb
Nov 10 at 11:19






Looks like you'd need to subclass JSONEncoder and override its encode and _make_iterencode methods to special-case instances of your class.
– snakecharmerb
Nov 10 at 11:19














@snakecharmerb Yes, I had a look at those methods but they don't seem to be designed to be extended so I'd end up duplicating most of the functionality of JSONEncoder :/
– Christian
Nov 12 at 14:37




@snakecharmerb Yes, I had a look at those methods but they don't seem to be designed to be extended so I'd end up duplicating most of the functionality of JSONEncoder :/
– Christian
Nov 12 at 14:37












2 Answers
2






active

oldest

votes

















up vote
2
down vote













just include the default parameter



def my_class_encoder(o):
if isinstance(o,MyClass):
return repr(o)

json.dumps(myClassInstance,default=my_class_encoder)


its easier to deal with than a real encoder



....



but really just add a def to your class



class MyPair(NamedTuple):
left: str
right: str
def serialize(self):
return list(self)
def __repr__(self):
return self.left + ':' + self.right


and then just



 json.dumps(myClassInstance.serialize())


this has the benefit of being more clear in what its doing (at least imho)






share|improve this answer





















  • Okay, my minimal example was probably to minimal. I of course still want to encode a proper JSON document that just includes MyPair objects somewhere in its tree.
    – Christian
    Nov 8 at 11:12


















up vote
0
down vote



accepted










So I ended up reimplementing a JSONEncoder more or less from scratch. Since I don't need any fancy pretty-printing, this is fairly straightforward:



class MyJSONEncoder(json.JSONEncoder):

def __init__(self, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False,
indent=None, separators=None, default=None):
super().__init__(skipkeys=skipkeys, ensure_ascii=ensure_ascii, check_circular=check_circular,
allow_nan=allow_nan, sort_keys=sort_keys, indent=indent, separators=separators,
default=default)
self._serializers: Set[Tuple[Type, Callable]] = {
(MyPair, lambda pair: '"' + str(pair) + '"',)
}

def default(self, o):
return super().default(o)

def encode(self, o):
return ''.join(self.iterencode(o))

def iterencode(self, o, _one_shot=False):
for t, serializer in self._serializers:
if isinstance(o, t):
yield serializer(o)
break
else:
if isinstance(o, bool):
yield "true" if o else "false"
elif isinstance(o, str):
yield '"' + o + '"'
elif isinstance(o, bytes):
yield '"' + o.decode("utf-8") + '"'
elif isinstance(o, int) or isinstance(o, float) or isinstance(o, Decimal):
yield str(o)
elif isinstance(o, Dict):
yield '{'
for num, (key, value) in enumerate(o.items()):
yield bool(num) * ', ' + '"' + str(key) + '": '
yield from self.iterencode(value)
yield '}'
elif isinstance(o, Sequence):
yield '['
for num, value in enumerate(o):
yield bool(num) * ', '
yield from self.iterencode(value)
yield ']'
else:
yield self.default(o)


For custom types, adding the type name and the function that stringifies it to self._serializers and you should be good. The iterencode() behaves differently from the normal one (mainly in that it yields the brackets separately and not alongside the first or last element) but I couldn't see where this would break anything.






share|improve this answer





















    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














     

    draft saved


    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53194044%2fhow-can-i-json-serialize-a-custom-iterable%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    2 Answers
    2






    active

    oldest

    votes








    2 Answers
    2






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    2
    down vote













    just include the default parameter



    def my_class_encoder(o):
    if isinstance(o,MyClass):
    return repr(o)

    json.dumps(myClassInstance,default=my_class_encoder)


    its easier to deal with than a real encoder



    ....



    but really just add a def to your class



    class MyPair(NamedTuple):
    left: str
    right: str
    def serialize(self):
    return list(self)
    def __repr__(self):
    return self.left + ':' + self.right


    and then just



     json.dumps(myClassInstance.serialize())


    this has the benefit of being more clear in what its doing (at least imho)






    share|improve this answer





















    • Okay, my minimal example was probably to minimal. I of course still want to encode a proper JSON document that just includes MyPair objects somewhere in its tree.
      – Christian
      Nov 8 at 11:12















    up vote
    2
    down vote













    just include the default parameter



    def my_class_encoder(o):
    if isinstance(o,MyClass):
    return repr(o)

    json.dumps(myClassInstance,default=my_class_encoder)


    its easier to deal with than a real encoder



    ....



    but really just add a def to your class



    class MyPair(NamedTuple):
    left: str
    right: str
    def serialize(self):
    return list(self)
    def __repr__(self):
    return self.left + ':' + self.right


    and then just



     json.dumps(myClassInstance.serialize())


    this has the benefit of being more clear in what its doing (at least imho)






    share|improve this answer





















    • Okay, my minimal example was probably to minimal. I of course still want to encode a proper JSON document that just includes MyPair objects somewhere in its tree.
      – Christian
      Nov 8 at 11:12













    up vote
    2
    down vote










    up vote
    2
    down vote









    just include the default parameter



    def my_class_encoder(o):
    if isinstance(o,MyClass):
    return repr(o)

    json.dumps(myClassInstance,default=my_class_encoder)


    its easier to deal with than a real encoder



    ....



    but really just add a def to your class



    class MyPair(NamedTuple):
    left: str
    right: str
    def serialize(self):
    return list(self)
    def __repr__(self):
    return self.left + ':' + self.right


    and then just



     json.dumps(myClassInstance.serialize())


    this has the benefit of being more clear in what its doing (at least imho)






    share|improve this answer












    just include the default parameter



    def my_class_encoder(o):
    if isinstance(o,MyClass):
    return repr(o)

    json.dumps(myClassInstance,default=my_class_encoder)


    its easier to deal with than a real encoder



    ....



    but really just add a def to your class



    class MyPair(NamedTuple):
    left: str
    right: str
    def serialize(self):
    return list(self)
    def __repr__(self):
    return self.left + ':' + self.right


    and then just



     json.dumps(myClassInstance.serialize())


    this has the benefit of being more clear in what its doing (at least imho)







    share|improve this answer












    share|improve this answer



    share|improve this answer










    answered Nov 7 at 17:15









    Joran Beasley

    71.4k676115




    71.4k676115












    • Okay, my minimal example was probably to minimal. I of course still want to encode a proper JSON document that just includes MyPair objects somewhere in its tree.
      – Christian
      Nov 8 at 11:12


















    • Okay, my minimal example was probably to minimal. I of course still want to encode a proper JSON document that just includes MyPair objects somewhere in its tree.
      – Christian
      Nov 8 at 11:12
















    Okay, my minimal example was probably to minimal. I of course still want to encode a proper JSON document that just includes MyPair objects somewhere in its tree.
    – Christian
    Nov 8 at 11:12




    Okay, my minimal example was probably to minimal. I of course still want to encode a proper JSON document that just includes MyPair objects somewhere in its tree.
    – Christian
    Nov 8 at 11:12












    up vote
    0
    down vote



    accepted










    So I ended up reimplementing a JSONEncoder more or less from scratch. Since I don't need any fancy pretty-printing, this is fairly straightforward:



    class MyJSONEncoder(json.JSONEncoder):

    def __init__(self, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False,
    indent=None, separators=None, default=None):
    super().__init__(skipkeys=skipkeys, ensure_ascii=ensure_ascii, check_circular=check_circular,
    allow_nan=allow_nan, sort_keys=sort_keys, indent=indent, separators=separators,
    default=default)
    self._serializers: Set[Tuple[Type, Callable]] = {
    (MyPair, lambda pair: '"' + str(pair) + '"',)
    }

    def default(self, o):
    return super().default(o)

    def encode(self, o):
    return ''.join(self.iterencode(o))

    def iterencode(self, o, _one_shot=False):
    for t, serializer in self._serializers:
    if isinstance(o, t):
    yield serializer(o)
    break
    else:
    if isinstance(o, bool):
    yield "true" if o else "false"
    elif isinstance(o, str):
    yield '"' + o + '"'
    elif isinstance(o, bytes):
    yield '"' + o.decode("utf-8") + '"'
    elif isinstance(o, int) or isinstance(o, float) or isinstance(o, Decimal):
    yield str(o)
    elif isinstance(o, Dict):
    yield '{'
    for num, (key, value) in enumerate(o.items()):
    yield bool(num) * ', ' + '"' + str(key) + '": '
    yield from self.iterencode(value)
    yield '}'
    elif isinstance(o, Sequence):
    yield '['
    for num, value in enumerate(o):
    yield bool(num) * ', '
    yield from self.iterencode(value)
    yield ']'
    else:
    yield self.default(o)


    For custom types, adding the type name and the function that stringifies it to self._serializers and you should be good. The iterencode() behaves differently from the normal one (mainly in that it yields the brackets separately and not alongside the first or last element) but I couldn't see where this would break anything.






    share|improve this answer

























      up vote
      0
      down vote



      accepted










      So I ended up reimplementing a JSONEncoder more or less from scratch. Since I don't need any fancy pretty-printing, this is fairly straightforward:



      class MyJSONEncoder(json.JSONEncoder):

      def __init__(self, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False,
      indent=None, separators=None, default=None):
      super().__init__(skipkeys=skipkeys, ensure_ascii=ensure_ascii, check_circular=check_circular,
      allow_nan=allow_nan, sort_keys=sort_keys, indent=indent, separators=separators,
      default=default)
      self._serializers: Set[Tuple[Type, Callable]] = {
      (MyPair, lambda pair: '"' + str(pair) + '"',)
      }

      def default(self, o):
      return super().default(o)

      def encode(self, o):
      return ''.join(self.iterencode(o))

      def iterencode(self, o, _one_shot=False):
      for t, serializer in self._serializers:
      if isinstance(o, t):
      yield serializer(o)
      break
      else:
      if isinstance(o, bool):
      yield "true" if o else "false"
      elif isinstance(o, str):
      yield '"' + o + '"'
      elif isinstance(o, bytes):
      yield '"' + o.decode("utf-8") + '"'
      elif isinstance(o, int) or isinstance(o, float) or isinstance(o, Decimal):
      yield str(o)
      elif isinstance(o, Dict):
      yield '{'
      for num, (key, value) in enumerate(o.items()):
      yield bool(num) * ', ' + '"' + str(key) + '": '
      yield from self.iterencode(value)
      yield '}'
      elif isinstance(o, Sequence):
      yield '['
      for num, value in enumerate(o):
      yield bool(num) * ', '
      yield from self.iterencode(value)
      yield ']'
      else:
      yield self.default(o)


      For custom types, adding the type name and the function that stringifies it to self._serializers and you should be good. The iterencode() behaves differently from the normal one (mainly in that it yields the brackets separately and not alongside the first or last element) but I couldn't see where this would break anything.






      share|improve this answer























        up vote
        0
        down vote



        accepted







        up vote
        0
        down vote



        accepted






        So I ended up reimplementing a JSONEncoder more or less from scratch. Since I don't need any fancy pretty-printing, this is fairly straightforward:



        class MyJSONEncoder(json.JSONEncoder):

        def __init__(self, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False,
        indent=None, separators=None, default=None):
        super().__init__(skipkeys=skipkeys, ensure_ascii=ensure_ascii, check_circular=check_circular,
        allow_nan=allow_nan, sort_keys=sort_keys, indent=indent, separators=separators,
        default=default)
        self._serializers: Set[Tuple[Type, Callable]] = {
        (MyPair, lambda pair: '"' + str(pair) + '"',)
        }

        def default(self, o):
        return super().default(o)

        def encode(self, o):
        return ''.join(self.iterencode(o))

        def iterencode(self, o, _one_shot=False):
        for t, serializer in self._serializers:
        if isinstance(o, t):
        yield serializer(o)
        break
        else:
        if isinstance(o, bool):
        yield "true" if o else "false"
        elif isinstance(o, str):
        yield '"' + o + '"'
        elif isinstance(o, bytes):
        yield '"' + o.decode("utf-8") + '"'
        elif isinstance(o, int) or isinstance(o, float) or isinstance(o, Decimal):
        yield str(o)
        elif isinstance(o, Dict):
        yield '{'
        for num, (key, value) in enumerate(o.items()):
        yield bool(num) * ', ' + '"' + str(key) + '": '
        yield from self.iterencode(value)
        yield '}'
        elif isinstance(o, Sequence):
        yield '['
        for num, value in enumerate(o):
        yield bool(num) * ', '
        yield from self.iterencode(value)
        yield ']'
        else:
        yield self.default(o)


        For custom types, adding the type name and the function that stringifies it to self._serializers and you should be good. The iterencode() behaves differently from the normal one (mainly in that it yields the brackets separately and not alongside the first or last element) but I couldn't see where this would break anything.






        share|improve this answer












        So I ended up reimplementing a JSONEncoder more or less from scratch. Since I don't need any fancy pretty-printing, this is fairly straightforward:



        class MyJSONEncoder(json.JSONEncoder):

        def __init__(self, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False,
        indent=None, separators=None, default=None):
        super().__init__(skipkeys=skipkeys, ensure_ascii=ensure_ascii, check_circular=check_circular,
        allow_nan=allow_nan, sort_keys=sort_keys, indent=indent, separators=separators,
        default=default)
        self._serializers: Set[Tuple[Type, Callable]] = {
        (MyPair, lambda pair: '"' + str(pair) + '"',)
        }

        def default(self, o):
        return super().default(o)

        def encode(self, o):
        return ''.join(self.iterencode(o))

        def iterencode(self, o, _one_shot=False):
        for t, serializer in self._serializers:
        if isinstance(o, t):
        yield serializer(o)
        break
        else:
        if isinstance(o, bool):
        yield "true" if o else "false"
        elif isinstance(o, str):
        yield '"' + o + '"'
        elif isinstance(o, bytes):
        yield '"' + o.decode("utf-8") + '"'
        elif isinstance(o, int) or isinstance(o, float) or isinstance(o, Decimal):
        yield str(o)
        elif isinstance(o, Dict):
        yield '{'
        for num, (key, value) in enumerate(o.items()):
        yield bool(num) * ', ' + '"' + str(key) + '": '
        yield from self.iterencode(value)
        yield '}'
        elif isinstance(o, Sequence):
        yield '['
        for num, value in enumerate(o):
        yield bool(num) * ', '
        yield from self.iterencode(value)
        yield ']'
        else:
        yield self.default(o)


        For custom types, adding the type name and the function that stringifies it to self._serializers and you should be good. The iterencode() behaves differently from the normal one (mainly in that it yields the brackets separately and not alongside the first or last element) but I couldn't see where this would break anything.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 14 at 13:08









        Christian

        270218




        270218






























             

            draft saved


            draft discarded



















































             


            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53194044%2fhow-can-i-json-serialize-a-custom-iterable%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            這個網誌中的熱門文章

            Academy of Television Arts & Sciences

            L'Équipe

            1995 France bombings