代码之家 › 专栏 › 技术社区 › krock1516

无法接收syslog日志。删除和替换函数的配置

elastic-stack

krock1516 · 技术社区 · 7 年前

我只是麋鹿的新手,尝试了一些测试,我能够运行一些测试,但当我尝试用 grok & mutate 删除(&M);替换我的syslog输出中的一些feild,我遇到以下错误。。

21:58:47.976 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, {, ,, ] at line 21, column 9 (byte 496) after filter {\n  if [type] == \"syslog\" {\n    grok {\n      match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\\[%{POSINT:pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n    }\n    date {\n      match => [ \"syslog_timestamp\", \"MMM  d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n    }\n    mutate {\n      remove_field => [\n        \"message\",\n        \"pid\",\n        \"port\"\n        "}

下面是我的配置文件。。。。

# cat logstash-syslog2.conf
input {
  file {
    path => [ "/scratch/rsyslog/*/messages.log" ]
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    mutate {
      remove_field => [
        "message",
        "pid",
        "port"
        "_grokparsefailure"
      ]
    }
    mutate {
      replace => [
        "@source_host", "%{allLogs_hostname}"
        "@message", "%{allLogs_message}"
      ]
    }
    mutate {
      remove => [
        "allLogs_hostname",
        "syslog_message",
        "syslog_timestamp"
      ]
    }
}
output {
  if [type] == "syslog" {
    elasticsearch {
      hosts => "localhost:9200"
      index => "%{type}-%{+YYYY.MM.dd}"
    }
  }
}

请建议我做错了什么,并帮助理解删除(&替换lagstash的函数。。

附言:我的麋鹿版本是5.4

2 回复 | 直到 7 年前

Ram 7 年前

您发布的配置有很多语法错误,logsatsh有自己的配置语言,并且希望配置文件遵守规则。这 link 具有完整的logstash配置语言参考。

我对你的配置文件做了一些更正并发布在这里,添加了我对配置文件本身错误的评论和解释

input 
{
    file 
    {
        path => [ "/scratch/rsyslog/*/messages.log" ]
        type => "syslog"
    }
}

filter 
{
    if [type] == "syslog" 
    {
        grok 
        {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}" }
        }

    date 
    {
        match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }

    # Have merged it with the remove_field option below
    #mutate {
    #  remove_field => [
    #    "message",
    #    "pid",
    #    "port",
    #    "_grokparsefailure"
    #  ]
    #}

    mutate 
    {

        # The replace option only accept hash data type which has a syntax as below 
        # For more details visit the below link
        # https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-replace
        replace => {
            "@source_host" => "%{allLogs_hostname}" 
            "@message" => "%{allLogs_message}"
        }
    }

    mutate 
    {
        # Mutate does not have remove option i guess your intention is to remove the event field
        # hence used remove_field option here
        # The remove_filed option only accepts arary as value type as shown below
        # For details read the below link
        # https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field
        remove_field => [
            "message",
            "pid",
            "port",
            "_grokparsefailure",
            "allLogs_hostname",
            "syslog_message",
            "syslog_timestamp"
        ]
    }
  }
}

output 
{
    if [type] == "syslog" 
    {
        elasticsearch 
        {
            # The Hosts option only takes uri as a value type , originally you have provided string as it's value type
            # For more info please read the below link
            #https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts
            hosts => ["localhost:9200"]
            index => "%{type}-%{+YYYY.MM.dd}"
        }
    }
}

您可以使用logstash命令行选项-t测试配置文件的语法是否正确。该选项将测试并报告配置文件的语法是否正确

bin\logstash -f 'path-to-your-config-file' -t

请让我知道任何澄清

ngi 7 年前

您必须在日志存储配置文件的“port”后面添加逗号。

   mutate {
      remove_field => [
        "message",
        "pid",
        "port",
        "_grokparsefailure"
      ]
    }