代码之家 › 专栏 › 技术社区 › Bruno Bronosky

如何使用jq使用嵌入的、转义的/字符串化的json格式化json(对于任何/所有结构)?

jq json javascript

Bruno Bronosky · 技术社区 · 5 年前

有很多问题说“我有这个确切的json结构,请帮我做功课,让它有另一个确切的结构。”这不是其中的一个问题。

我想看 jq 用于清理通用数据的脚本。我将在下面发布一个答案,它对我从日志中获得的大多数数据都非常有效。但是,鉴于这里有很多聪明人,我想看看其他人有什么想法。

注: 这并不是其他问题的重复,因为我在上面看到的每个问题的性质都会得到答案,而这些答案只对操作人员有用。这个问题的答案应该有助于每个搜索类似问题的人。一些幸运的回答者会得到名誉炸弹 like I did with my answer here .

2 回复 | 直到 5 年前

peak 5 年前

运行 fromjson 所有字符串都可以完成,例如使用 walk 和 ? :

walk(if type == "string" then fromjson? // . else . end)

只留下数字字符串

walk(if type == "string"
     then if tonumber? // false then . 
          else fromjson? // . end
     else . end)

Bruno Bronosky 5 年前

假设所有嵌入的json都是一个缩小的字典(就像我的terraform输出、aws cli输出和aws日志一样) jq 剧本能创造奇迹。

jq 'walk(if type == "string" and .[0:2] == "{\"" then .=(.|fromjson) else . end)'

它通过遍历json对象来查找以 {" 并使用子流程将它们通过管道 fromjson (永不离开 JQ )

我把它放在一个bash函数中( jqp )因为它比为别名转义引号更容易,而且更灵活。然后我可以用它来处理文件或剪贴板的内容。

# This is in my .bash_profile
jqp(){
  jq 'walk(if type == "string" and .[0:2] == "{\"" then .=(.|fromjson) else . end)' "$@"
}

# Here is an event trigger from SNS to Lambda
$ cat event.json
{
    "Records": [
        {
            "EventVersion": "1.0",
            "EventSubscriptionArn": "arn:aws:sns:us-east-1:123456789012:sns-to-slack-shared-services:a70df027-2c7f-492a-840a-633d44fd71a6",
            "EventSource": "aws:sns",
            "Sns": {
                "SignatureVersion": "1",
                "Timestamp": "2019-02-06T15:50:30.028Z",
                "Signature": "GN3712/aWjVLftSzdOcW5Zm32/uvfZKrCcvTmz6Obv/AXbz1xc22sTMYt2vFja7coHGhhO5bG6dz/IbJSx/Zm0U/dVVefWKukFl1umP3av+1JoUbbi+4uHai3k3AwQa3wR4HWjrKKmMt+Tkt/gm7jvhcuojtx+n5oc4S6bMsVq5OmSfAWd2Xd1urTm2zeGCL59nbfhZv+xB4db3dk62FtxVKtFXtvO2pH27+E3vXUvgu2k1c2Kd/Vt/vbYCAA==",
                "SigningCertUrl": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-a70df027-2c7f-492a-840a-633d44fd71a6.pem",
                "MessageId": "a8df3067-c347-55ce-b869-64b2c7c1d0a3",
                "Message": "{\"AlarmName\":\"unauthorized_api_calls_Count-alarm\",\"AlarmDescription\":\"This metric monitors unauthorized API calls\",\"AWSAccountId\":\"123456789012\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 datapoint [5.0 (06/02/19 15:45:00)] was greater than or equal to the threshold (1.0).\",\"StateChangeTime\":\"2019-02-06T15:50:30.023+0000\",\"Region\":\"US East (N. Virginia)\",\"OldStateValue\":\"INSUFFICIENT_DATA\",\"Trigger\":{\"MetricName\":\"unauthorized_api_calls\",\"Namespace\":\"security_rules\",\"StatisticType\":\"Statistic\",\"Statistic\":\"SUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanOrEqualToThreshold\",\"Threshold\":1.0,\"TreatMissingData\":\"\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
                "MessageAttributes": {},
                "Type": "Notification",
                "UnsubscribeUrl": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789012:sns-to-slack-shared-services:a70df027-2c7f-492a-840a-633d44fd71a6",
                "TopicArn": "arn:aws:sns:us-east-1:123456789012:sns-to-slack-shared-services",
                "Subject": "ALARM: \"unauthorized_api_calls_Count-alarm\" in US East (N. Virginia)"
            }
        }
    ]
}

# Demonstrate that "$@" in the function allows the use of extra options
$ jqp --indent 4 event.json
{
    "Records": [
        {
            "EventVersion": "1.0",
            "EventSubscriptionArn": "arn:aws:sns:us-east-1:123456789012:sns-to-slack-shared-services:a70df027-2c7f-492a-840a-633d44fd71a6",
            "EventSource": "aws:sns",
            "Sns": {
                "SignatureVersion": "1",
                "Timestamp": "2019-02-06T15:50:30.028Z",
                "Signature": "GN3712/aWjVLftSzdOcW5Zm32/uvfZKrCcvTmz6Obv/AXbz1xc22sTMYt2vFja7coHGhhO5bG6dz/IbJSx/Zm0U/dVVefWKukFl1umP3av+1JoUbbi+4uHai3k3AwQa3wR4HWjrKKmMt+Tkt/gm7jvhcuojtx+n5oc4S6bMsVq5OmSfAWd2Xd1urTm2zeGCL59nbfhZv+xB4db3dk62FtxVKtFXtvO2pH27+E3vXUvgu2k1c2Kd/Vt/vbYCAA==",
                "SigningCertUrl": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-a70df027-2c7f-492a-840a-633d44fd71a6.pem",
                "MessageId": "a8df3067-c347-55ce-b869-64b2c7c1d0a3",
                "Message": {
                    "AlarmName": "unauthorized_api_calls_Count-alarm",
                    "AlarmDescription": "This metric monitors unauthorized API calls",
                    "AWSAccountId": "123456789012",
                    "NewStateValue": "ALARM",
                    "NewStateReason": "Threshold Crossed: 1 datapoint [5.0 (06/02/19 15:45:00)] was greater than or equal to the threshold (1.0).",
                    "StateChangeTime": "2019-02-06T15:50:30.023+0000",
                    "Region": "US East (N. Virginia)",
                    "OldStateValue": "INSUFFICIENT_DATA",
                    "Trigger": {
                        "MetricName": "unauthorized_api_calls",
                        "Namespace": "security_rules",
                        "StatisticType": "Statistic",
                        "Statistic": "SUM",
                        "Unit": null,
                        "Dimensions": [],
                        "Period": 300,
                        "EvaluationPeriods": 1,
                        "ComparisonOperator": "GreaterThanOrEqualToThreshold",
                        "Threshold": 1,
                        "TreatMissingData": "",
                        "EvaluateLowSampleCountPercentile": ""
                    }
                },
                "MessageAttributes": {},
                "Type": "Notification",
                "UnsubscribeUrl": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:123456789012:sns-to-slack-shared-services:a70df027-2c7f-492a-840a-633d44fd71a6",
                "TopicArn": "arn:aws:sns:us-east-1:123456789012:sns-to-slack-shared-services",
                "Subject": "ALARM: \"unauthorized_api_calls_Count-alarm\" in US East (N. Virginia)"
            }
        }
    ]
}