代码之家  ›  专栏  ›  技术社区  ›  Goku

Python:将字符串输出格式化为JSON

  •  0
  • Goku  · 技术社区  · 1 年前

    我正在尝试使用FastAPI和Structlog,并想测试日志格式,将其从纯文本/字符串转换为JSON格式,以获得更好的可读性和日志聚合平台的处理能力。面对这样一种情况:某些日志输出以JSON形式可用,但以纯字符串形式存在。

    电流输出

    INFO:     127.0.0.1:62154 - "GET /api/preface HTTP/1.1" 200 OK
    INFO:     127.0.0.1:62154 - "GET /loader.json HTTP/1.1" 200 OK
    INFO:     127.0.0.1:62155 - "GET /hello_world HTTP/1.1" 200 OK
    {"key":"test_key","message":"Push to NFS Success","event":"Testing Fast API..","logger":"test_my_api","filename":"main.py","func_name":"Hello_World","process":23760,"module":"docker","thread":23140,"pathname":"D:\\my_work\\fast_api\\main.py","process_name":"SpawnProcess-1","level":"info","time-iso":"2023-06-30T15:25:03.113400Z"}
    

    预期输出:

        {
        "level": "INFO",
        "IP": "127.0 .0 .1: 62154",
        "method": "GET",
        "endpoint": "/loader.json",
        "protocol": "HTTP / 1.1",
        "status_code": 200,
        "status": "OK"
    }
        {
        "level": "INFO",
        "IP": "127.0 .0 .1: 62155",
        "method": "GET",
        "endpoint": "/api/preface",
        "protocol": "HTTP / 1.1",
        "status_code": 200,
        "status": "OK"
    }
    
     {
        "level": "INFO",
        "IP": "127.0 .0 .1: 62155",
        "method": "GET",
        "endpoint": "/hello_world",
        "protocol": "HTTP / 1.1",
        "status_code": 200,
        "status": "OK"
    }
        {"key":"test_key","message":"Push to NFS Success","event":"Testing Fast API..","logger":"test_my_api","filename":"main.py","func_name":"Hello_World","process":23760,"module":"docker","thread":23140,"pathname":"D:\\my_work\\fast_api\\main.py","process_name":"SpawnProcess-1","level":"info","time-iso":"2023-06-30T15:25:03.113400Z"}
    

    我在这里错过了什么?谢谢

    结构.py

    import orjson
    import structlog
    import logging
    
    ## Added only the necessary context.
    class StructLogTest:
        def __init__(self, logging_level=logging.DEBUG, logger_name="test"):
            self.logging_level = logging_level
            self.logger_name = logger_name
            StructLogTest.logger_name_var = self.logger_name
            self.configure_structlog(self.logging_level, self.logger_name)
    
        def logger_name(_, __, event_dict):
            event_dict["test_log"] = StructLogTest.logger_name_var
            return event_dict
    
    
        @staticmethod
        def configure_structlog(logging_level, logger_name):
            structlog.configure(
                processors=[
                    StructLogTest.logger_name,
                    structlog.threadlocal.merge_threadlocal,
                    structlog.processors.CallsiteParameterAdder(),
                    structlog.processors.add_log_level,
                    structlog.stdlib.PositionalArgumentsFormatter(),
                    structlog.processors.StackInfoRenderer(),
                    structlog.processors.format_exc_info,
                    structlog.processors.TimeStamper(fmt="iso", utc=True, key="time-iso"),
                    structlog.processors.JSONRenderer(serializer=orjson.dumps),
                ],
                wrapper_class=structlog.make_filtering_bound_logger(logging_level),
                context_class=dict,
                logger_factory=structlog.BytesLoggerFactory(),
            )
            return structlog
    
        def define_Logger(self, *args, **kwargs):
            return structlog.get_logger(*args, **kwargs)
    
        def info(self, message, *args, **kwargs):
            return structlog.get_logger().info(message, *args, **kwargs)
        
        and other methods so on..
    
    
        
    

    主要.py

    from struct import StructLogTest
    from fastapi import APIRouter
    import requests
    from requests.auth import HTTPBasicAuth
    from requests import Response
    
    log = StructLogTest(logger_name="test_my_api")
    log = log.get_Logger()
    
    @router.get("/hello_world")
    def Hello_World():
        logg = log.bind(key=test_key)
        logg.info(
            "Testing Fast API..",
            message=some_other_meaningful_function.dump(),
        )
        return {" Hello World !! "}
    
    0 回复  |  直到 1 年前
        1
  •  1
  •   wim    1 年前

    Structlog是一个与stdlib完全独立的日志框架 logging 。它不会为您配置stdlib日志记录框架。 Uvicorn uses stdlib logging ,并且您不能改变这一点,除非对源代码进行分叉和编辑以使用其他日志记录框架。

    不过,你确实有一些选择。最简单的方法就是使用json格式化程序配置stdlib日志记录,例如create uvicorn-logconfig.ini 以下为:

    [loggers]
    keys=root, uvicorn, gunicorn
    
    [handlers]
    keys=access_handler
    
    [formatters]
    keys=json
    
    [logger_root]
    level=INFO
    handlers=access_handler
    propagate=1
    
    [logger_gunicorn]
    level=INFO
    handlers=access_handler
    propagate=0
    qualname=gunicorn
    
    [logger_uvicorn]
    level=INFO
    handlers=access_handler
    propagate=0
    qualname=uvicorn
    
    [handler_access_handler]
    class=logging.StreamHandler
    formatter=json
    args=()
    
    [formatter_json]
    class=pythonjsonlogger.jsonlogger.JsonFormatter
    

    它使用来自 python-json-logger 。现在,使用以下内容运行您的应用程序:

    uvicorn --log-config=uvicorn-logconfig.ini main:router
    

    来自uvicorn和您自己的应用程序代码的日志将呈现为JSON,前者通过stdlib格式器,后者通过structlog处理器。

    以下是我在启动您的应用程序并向 http://127.0.0.1:8000/hello_world 从structlog获取“Testing Fast API…”事件,然后用Ctrl+C关闭uvicorn:

    $ uvicorn --log-config=uvicorn-logconfig.ini main:router
    {"message": "Started server process [54894]", "color_message": "Started server process [\u001b[36m%d\u001b[0m]"}
    {"message": "Waiting for application startup."}
    {"message": "Application startup complete."}
    {"message": "Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)", "color_message": "Uvicorn running on \u001b[1m%s://%s:%d\u001b[0m (Press CTRL+C to quit)"}
    {"test_key":"test_val","message":"some_message","event":"Testing Fast API..","pathname":"/private/tmp/s/main.py","lineno":42,"process_name":"MainProcess","func_name":"Hello_World","filename":"main.py","thread_name":"AnyIO worker thread","thread":12930912256,"module":"main","process":54894,"level":"info","time-iso":"2023-07-02T19:22:58.085382Z"}
    {"message": "127.0.0.1:58519 - \"GET /hello_world HTTP/1.1\" 200"}
    ^C{"message": "Shutting down"}
    {"message": "Waiting for application shutdown."}
    {"message": "Application shutdown complete."}
    {"message": "Finished server process [54894]", "color_message": "Finished server process [\u001b[36m%d\u001b[0m]"}
    

    这被推荐为structlog文档中最简单的方法(参见小节 "Don't Integrate" 在“标准库”部分中):

    最直接的选项是配置标准库 测井 接近什么 结构日志 正在登录并将其留在那里。

    由于这些通常是来自第三方的日志条目,不会利用 结构日志 令人惊讶的是,这通常是一种非常合适的方法。

    例如,如果您在生产中记录JSON,请将日志记录配置为使用 python-json-logger 让它也打印JSON,然后调整配置以匹配它们的输出。

    由于stdlib日志记录是高度可配置的,structlog也非常灵活,因此还有其他方法可以更紧密地集成structlog和stdlib。看见 Rendering Using structlog -based Formatters Within logging 如果感兴趣的话。