代码之家 › 专栏 › 技术社区 › Kyle Bridenstine

Airflow EC2实例socket.getfqdn()错误

airflow-scheduler hostname airflow ip amazon-ec2

Kyle Bridenstine · 技术社区 · 6 年前

我使用的是Airflow版本1.9,他们的软件中有一个bug,您可以阅读 here on my previous Stackoverflow post here on another one of my Stackoverflow posts ,和 here on Airflow's Github where the bug is reported and discussed .

socket.getfqdn()

问题是,在Amazon EC2实例(Amazon Linux 1)上,此命令不返回IP地址,而是返回如下所示的主机名:

IP-1-2-3-4

1.2.3.4

获取我从中找到的IP值 here 我可以使用这个命令:

socket.gethostbyname(socket.gethostname())

我已经在Python shell中测试了这个命令,它返回了正确的值。所以我对气流包进行了搜索 socket.getfqdn() 这就是我得到的:

[airflow@ip-1-2-3-4 site-packages]$ cd airflow/
[airflow@ip-1-2-3-4 airflow]$ grep -r "fqdn" .

./security/utils.py:    fqdn = host
./security/utils.py:    if not fqdn or fqdn == '0.0.0.0':
./security/utils.py:        fqdn = get_localhost_name()
./security/utils.py:    return '%s/%s@%s' % (components[0], fqdn.lower(), components[2])
./security/utils.py:    return socket.getfqdn()
./security/utils.py:def get_fqdn(hostname_or_ip=None):
./security/utils.py:            fqdn = socket.gethostbyaddr(hostname_or_ip)[0]
./security/utils.py:            fqdn = get_localhost_name()
./security/utils.py:        fqdn = hostname_or_ip
./security/utils.py:    if fqdn == 'localhost':
./security/utils.py:        fqdn = get_localhost_name()
./security/utils.py:    return fqdn

Binary file ./security/__pycache__/utils.cpython-36.pyc matches
Binary file ./security/__pycache__/kerberos.cpython-36.pyc matches

./security/kerberos.py:    principal = configuration.get('kerberos', 'principal').replace("_HOST", socket.getfqdn())
./security/kerberos.py:        principal = "%s/%s" % (configuration.get('kerberos', 'principal'), socket.getfqdn())

Binary file ./contrib/auth/backends/__pycache__/kerberos_auth.cpython-36.pyc matches

./contrib/auth/backends/kerberos_auth.py:        service_principal = "%s/%s" % (configuration.get('kerberos', 'principal'), utils.get_fqdn())

./www/views.py:            'airflow/circles.html', hostname=socket.getfqdn()), 404
./www/views.py:            hostname=socket.getfqdn(),

Binary file ./www/__pycache__/app.cpython-36.pyc matches
Binary file ./www/__pycache__/views.cpython-36.pyc matches

./www/app.py:                'hostname': socket.getfqdn(),

Binary file ./__pycache__/jobs.cpython-36.pyc matches
Binary file ./__pycache__/models.cpython-36.pyc matches

./bin/cli.py:    hostname = socket.getfqdn()

Binary file ./bin/__pycache__/cli.cpython-36.pyc matches

./config_templates/default_airflow.cfg:# gets augmented with fqdn

./jobs.py:        self.hostname = socket.getfqdn()
./jobs.py:        fqdn = socket.getfqdn()
./jobs.py:        same_hostname = fqdn == ti.hostname
./jobs.py:                                "{fqdn}".format(**locals()))

Binary file ./api/auth/backend/__pycache__/kerberos_auth.cpython-36.pyc matches

./api/auth/backend/kerberos_auth.py:from socket import getfqdn
./api/auth/backend/kerberos_auth.py:        hostname = getfqdn()

./models.py:        self.hostname = socket.getfqdn()
./models.py:        self.hostname = socket.getfqdn()

我不确定是否应该替换 socket.getfqdn() socket.gethostbyname(socket.gethostname()) 或者不。其中之一,这将是麻烦的维护,因为我将不再使用气流包,我从Pip安装。我试着升级到Airflow1.10版本,但它非常有缺陷,我无法启动和运行。所以现在我似乎还停留在Airflow1.9版本上,但我需要纠正这个AirflowBug,因为它导致我的任务偶尔失败。

1 回复 | 直到 6 年前

Kyle Bridenstine 5 年前

只需将所有发生错误的函数调用替换为有效的函数调用。下面是我跑的步骤。如果使用气流群集,请确保对所有气流服务器(主服务器和工作服务器)执行此操作。

[ec2-user@ip-1-2-3-4 ~]$ cd /usr/local/lib/python3.6/site-packages/airflow

[ec2-user@ip-1-2-3-4 airflow]$ grep -r "socket.getfqdn()" .
./security/utils.py:    return socket.getfqdn()
./security/kerberos.py:    principal = configuration.get('kerberos', 'principal').replace("_HOST", socket.getfqdn())
./security/kerberos.py:        principal = "%s/%s" % (configuration.get('kerberos', 'principal'), socket.getfqdn())
./www/views.py:            'airflow/circles.html', hostname=socket.getfqdn()), 404
./www/views.py:            hostname=socket.getfqdn(),
./www/app.py:                'hostname': socket.getfqdn(),
./bin/cli.py:    hostname = socket.getfqdn()
./jobs.py:        self.hostname = socket.getfqdn()
./jobs.py:        fqdn = socket.getfqdn()
./models.py:        self.hostname = socket.getfqdn()
./models.py:        self.hostname = socket.getfqdn()

[ec2-user@ip-1-2-3-4 airflow]$ sudo find . -type f -exec sed -i 's/socket.getfqdn()/socket.gethostbyname(socket.gethostname())/g' {} +

[ec2-user@ip-1-2-3-4 airflow]$ grep -r "socket.getfqdn()" .

[ec2-user@ip-1-2-3-4 airflow]$ grep -r "socket.gethostbyname(socket.gethostname())" .

./security/utils.py:    return socket.gethostbyname(socket.gethostname())
./security/kerberos.py:    principal = configuration.get('kerberos', 'principal').replace("_HOST", socket.gethostbyname(socket.gethostname()))
./security/kerberos.py:        principal = "%s/%s" % (configuration.get('kerberos', 'principal'), socket.gethostbyname(socket.gethostname()))
./www/views.py:            'airflow/circles.html', hostname=socket.gethostbyname(socket.gethostname())), 404
./www/views.py:            hostname=socket.gethostbyname(socket.gethostname()),
./www/app.py:                'hostname': socket.gethostbyname(socket.gethostname()),
./bin/cli.py:    hostname = socket.gethostbyname(socket.gethostname())
./jobs.py:        self.hostname = socket.gethostbyname(socket.gethostname())
./jobs.py:        fqdn = socket.gethostbyname(socket.gethostname())
./models.py:        self.hostname = socket.gethostbyname(socket.gethostname())
./models.py:        self.hostname = socket.gethostbyname(socket.gethostname())

完成更新后,只需重新启动Airflow Web服务器、调度程序和工作进程,就可以完成所有设置。注意,当我使用python3.6开发airlow的python包时,我使用的是python3.6,有些人可能使用的是like 3.7,所以您的路径可能需要调整为like/usr/local/lib/python3.7/site-packages/airlow,所以只需cd到/usr/local/lib中,然后查看您需要进入哪个python文件夹。我不认为气流在这个位置下流动,但有时python包也在这里/usr/local/ /python3.6/site-packages的路径不同之处在于它是lib64而不是lib。另外,请记住,这在Airflow 1.10版中是固定的,因此您不必再在Airflow的最新版本中进行这些更改。