Wsrtjtyk

I am trying to collect specific information from very large log files but cannot figure out how to get the behavior I need.

For reference, an example log is sort of like this:

garbage I don't need - garbage I don't need

timestamp - date - server info - 'keyword 1' - data

more data more data more data more data

more data more data more data more data

more data more data 'keyword 2' - last bit of data

garbage I don't need - garbage I don't need

What I need is to find 'keyword 1', grab the whole line keyword 1 is on (back to timestamp) and all subsequent lines until (and including) the whole line that 'keyword 2' is on (through the last bit of data).

So far I have tried a few things. I cannot get decent results with re methods (findall, match, search etc.); I cannot figure out how to grab the data before the match (even with a look behind) but more importantly, I cannot figure out how to have the capture stop at a phrase and not just a single character.

for match in re.findall('keyword1[keyword2]+|', showall.read()):

I also tried something like this:

start_capture = False

for current_line in fileName:

    if 'keyword1' in current_line:

        start_capture = True

    if start_capture:

        new_list.append(current_line)

    if 'keyword2' in current_line:

        return(new_list)

No matter what I tried, this returned an empty list

Finally,I tried something like this:

def takewhile_plus_next(predicate, xs):

for x in xs:

    if not predicate(x):

        break

    yield x

yield x

with lastdb as f:

    lines = map(str.rstrip, f)

    skipped = dropwhile(lambda line: 'Warning: fatal assert' not in line, lines)

    lines_to_keep = takewhile_plus_next(lambda line: 'uptime:' not in line, skipped)

This last one took everything from keyword 1 to the EOF which includes almost 100,000 lines of garbage data.

You can use regex if you specify re.dotall and use lazy anythings .*? to match start and end:

import re



regex = r"n.*?(keyword 1).*?(keyword 2).*?$"



test_str = ("garbage I don't need - garbage I don't needn"

    "timestamp - date - server info - 'keyword 1' - datan"

    "more data more data more data more datan"

    "more data more data more data more datan"

    "more data more data 'keyword 2' - last bit of datan"

    "garbage I don't need - garbage I don't need")



matches = re.finditer(regex, test_str, re.DOTALL | re.MULTILINE)



for matchNum, match in enumerate(matches):

    matchNum = matchNum + 1



    print (match.group()) # your match is the whole group

Output:

timestamp - date - server info - 'keyword 1' - data 

more data more data more data more data

more data more data more data more data

more data more data 'keyword 2' - last bit of data

You might need to strip('n') from it ...

You can view it here: https://regex101.com/r/HWIALZ/1 - it also holds the explanation of the patter. The short of it:

n        newline 

   .*?    as few as possible anythings

   (keyword 1)   literal text - the () are not needed only if you want the group

   .*?    as few as possible anythings

   (keyword 2)   literal text - again () are not needed 

   .*?    as few as possible anythings

$         end of line

I included the () for clarity - you do not evaluate groups, you you remove them.

The following is fast for any size of file. It extracts from a 250M log file of nearly 2 million lines in 3 seconds. The extracted portion was at the end of the file.

I would not recommend using list, regexes or other in-memory techniques if there is a chance your files won't fit in available memory.

Test text file startstop_text:

line 1 this should not appear in output

line 2 keyword1

line 3 appears in output

line 4 keyword2

line 5 this should not appear in output

Code:

from itertools import dropwhile





def keepuntil(contains_end_keyword, lines):

    for line in lines:

        yield line

        if contains_end_keyword(line):

            break





with open('startstop_text', 'r') as f:

    from_start_line = dropwhile(lambda line: 'keyword1' not in line, f)

    extracted = keepuntil(lambda line: 'keyword2' in line, from_start_line)

    for line in extracted:

        print(line.rstrip())





>>> python startstop.py

line 2 keyword1

line 3 appears in output

line 4 keyword2

None of the other responses worked but I was able to figure it out using regex.

for match in re.findall(".*keyword1[sS]*?keyword2:[sS]*?keyword3.*", log_file.read()):