Wsrtjtyk

I'm attempting to create a parser to translate a "custom" file into JSON so I can more easily manipulate its contents (For argument's sake, call the "custom" formate a .qwerty).

I've already created a Lexer which breaks down the file into individual lexemes (tokens) which structure is [token_type, token_value]. Now I am struggling to parse the lexemes into their correct dictionaries, as it is difficult to insert data into a sub-sub-dictionary since Keys aren't constant. As well as insert data into arrays stored in dictionaries.

It should be noted I am attempting to sequentially parse tokens into an actual python json object then dump the json object.

An example of the file can be seen below, along with what the end result is meant to resemble.

FILE: ABC.querty

Dict_abc_1{



    Dict_abc_2{

        HeaderGUID="";

        Version_TPI="999";

        EncryptionType="0";

    }



    Dict_abc_3{

        FamilyName="John Doe";

    }



    Dict_abc_4{

        Array_abc{

            {TimeStamp="2018-11-07 01:00:00"; otherinfo="";}

            {TimeStamp="2018-11-07 01:00:00"; otherinfo="";}

            {TimeStamp="2018-11-07 01:00:00"; otherinfo="";}

            {TimeStamp="2018-11-07 02:53:57"; otherinfo="";}

            {TimeStamp="2018-11-07 02:53:57"; otherinfo="";}

        }



        Dict_abc_5{

            LastContact="2018-11-08 01:00:00";

            BatteryStatus=99;

            BUStatus=PowerOn;

            LastCallTime="2018-11-08 01:12:46";

            LastSuccessPoll="2018-11-08 01:12:46";

            CallResult=Successful;

        }

    }

}

Code=999999;

FILE: ABC.json

{  

    "Dict_abc_1":{

        "Dict_abc_2":{

            "HeaderGUID":"",

            "Version_TPI":"999",

            "EncryptionType":"0"

        },



        "Dict_abc_3":{

            "FamilyName":"John Doe"

        },



        "Dict_abc_4":{

            "Array_abc":[

                {"TimeStamp":"2018-11-07 01:00:00", "otherinfo":""},

                {"TimeStamp":"2018-11-07 01:00:00", "otherinfo":""},

                {"TimeStamp":"2018-11-07 01:00:00", "otherinfo":""},

                {"TimeStamp":"2018-11-07 02:53:57", "otherinfo":""},

                {"TimeStamp":"2018-11-07 02:53:57", "otherinfo":""}

            ],



            "Dict_abc_5":{

                "LastContact":"2018-11-08 01:00:00",

                "BatteryStatus":99,

                "BUStatus":"PowerOn",

                "LastCallTime":"2018-11-08 01:12:46",

                "LastSuccessPoll":"2018-11-08 01:12:46",

                "CallResult":"Successful"

            }

        }

    },

    "Code":999999

}

Additional token information,
Token types can either be (with possible values)

• 
  IDENTIFIER contain the name of the variable identifier


• 
  VARIABLE containing actual data belonging to the parent IDENTIFIER


• 
  OPERATOR equal "="


• 
  OPEN_BRACKET equal "{"


• 
  CLOSE_BRACKET equal "}"

An example of ABC.querty's lexemes can be seen HERE

fundamental logical extract of main.py

def main():

    content = open_file(file_name) ## read file



    lexer = Lexer(content) ## create lexer class

    tokens = lexer.tokenize() ## create lexems as seen in pastebin



    parser = Parser(tokens).parse() ## create parser class given tokens 



    print(json.dumps(parser, sort_keys=True,indent=4, separators=(',', ': ')))

parser.py

import re



class Parser(object):

    def __init__(self, tokens):

        self.tokens = tokens

        self.token_index = 0

        self.json_object = {}

        self.current_object = {}

        self.path = [self.json_object]



    def parse(self):

        while self.token_index < len(self.tokens):



            token = self.getToken()

            token_type = token[0]

            token_value = token[1]



            print("%s t %s" % (token_type, token_value))



            if token_type in "IDENTIFIER":

                self.increment()

                identifier_type = self.getToken()

                if identifier_type[0] in "OPEN_BRACKET":

                    identifier_two_type = self.getToken(1)

                    if identifier_two_type[0] in ["OPERATOR","IDENTIFIER"]:

                        ## make dict in current dict 

                        pass



                    elif identifier_two_type[0] in "OPEN_BRACKET":

                        ## make array in current dict 

                        pass



                elif identifier_type[0] in "OPERATOR":

                    ## insert data into current dict

                    pass





            if token_type in "CLOSE_BRACKET":

                identifier_type = self.getToken()

                if "OPEN_BRACKET" in identifier_type[0]:

                    #still in array of current dict

                    pass

                elif "IDENTIFIER" in identifier_type[0]:

                    self.changeDirectory()



                else:

                    #end script

                    pass



            self.increment()

        print(self.path)

        return self.json_object





    def changeDirectory(self):

        if len(self.path) > 0:

            self.path = self.path.pop()

            self.current_object = -1



    def increment(self):

        if self.token_index < len(self.tokens):

            self.token_index+=1



    def getToken(self, x=0):

        return self.tokens[self.token_index+x]

Additional parse information,
Currently, I was trying to store the current dictionary in a path array to allow me to insert into dictionaries and arrays within dictionaries.

Any suggestions or solutions are very much appreciated,

Thanks.

Last time I solved this problem I find out that finite-state machine is very helpful. I want to recommend the way after you have tokens but I don't know how it's called in english. The principle is: you go through tokens and add one by one on stack. After adding on stack you are checking stack for some rules. Like you combine primitive tokens into expressions that might be a part of more complex expressions.

For example "FamilyName":"John Doe". Tokens are "FamilyName", : and "John Doe".

You add first token on stack.
stack = ["FamilyName"].
Rule 1: str_obj -> E. So you create Expression(type='str', value="FamilyName") and stack is now stack = [Expression].

Then you add next token.
stack = [Expression, ':']. No rules for ':'. Go next.

stack = [Expression, ':', "FamilyName"]. Again we meet rule 1. So stack becomes stack = [Expression, ':', Expression]. Then we see another rule. Rule 2: E:E -> E. Use it like Expression(type='kv_pair, value=(Expression, Expression)). And stack becomes stack=[Expression].

And if you describes all the rules it will work like that. Hope it helps.