熊猫读取日志文件

感谢@edouardtheron&在正确方向上的推动;模块 shlex 。

如果您有更好的解决方案,请随时回答

但我想到的是,首先,导入库:

import shlex
import pandas as pd

创建一些示例数据:

# Example data
test_string = """
Tue Apr  3 08:51:05 2018 foo=123 bar=321 spam=eggs msg="String with spaces in it"
Tue Apr  3 10:31:46 2018 foo=111 bar=222 spam=eggs msg="Different string"
"""

创建与整行匹配但将其分组的正则表达式

1: 起始日期 ((?:[a-zA-Z]{3,4} ){2} \d \d\d:\d\d:\d\d \d{4})

2: 其他一切 (.*)

patt = re.compile('((?:[a-zA-Z]{3,4} ){2} \d \d\d:\d\d:\d\d \d{4}) (.*)')

循环测试字符串中的行并应用regex,然后解析 key_values 输入字典,使用 shlex公司

sers = []
for line in test_string.split('\n'):

    matt = re.match(patt, line)
    if not matt:
        # skip the empty lines
        continue
    # Extract Groups
    time, key_values = matt.groups()

    ser = pd.Series(dict(token.split('=', 1) for token in shlex.split(key_values)))
    ser['log_time'] = time
    sers.append(ser)

最后,将所有行连接到一个数据帧中:

# Concat serieses into a dataframe
df = pd.concat(sers, axis=1).T
# Change the type of 'log_time' to an actual date
df['log_time'] = pd.to_datetime(df['log_time'], format='%a %b  %d %X %Y', exact=True)

这将生成以下数据帧:

   bar  foo                       msg  spam            log_time
0  321  123  String with spaces in it  eggs 2018-04-03 08:51:05
1  222  111          Different string  eggs 2018-04-03 10:31:46