代码之家 › 专栏 › 技术社区 › Ari

使用AWS lambda运行python脚本,如何保存数据?

aws-lambda sqlite amazon-web-services python

Ari · 技术社区 · 5 年前

我有一个从API收集数据的脚本,在本地机器上手动运行这个脚本,我可以将数据保存到csv或sqlite.db文件中。

如果我把它放在AWS lambda上,如何存储和检索数据?

3 回复 | 直到 5 年前

Onema Swaminathan S 5 年前

/tmp see lambda limits

# lambda_function.py
import os
import boto3
from botocore.client import Config
import requests

s3 = boto3.resource('s3')
client = boto3.client('s3', config=Config(signature_version='s3v4'))

# This environment variable is set via the serverless.yml configuration
bucket = os.environ['FILES_BUCKET']

def lambda_handler(event, conntext):
    # Make the API CALL
    response = requests.get('https://google.com')

    # Get the data you care and transform it to the desire format
    body = response.text

    # Save it to local storage
    tmp_file_path = "/tmp/website.html"
    with open(tmp_file_path, "w") as file:
        file.write(body)
    s3.Bucket(bucket).upload_file(tmp_file_path, 'website.html')

    # OPTIONAL: Generar signed URL to download the file
    url = client.generate_presigned_url(
        ClientMethod='get_object',
        Params={
            'Bucket': bucket,
            'Key': 'website.html'
        },
        ExpiresIn=604800 # 7 days
    )
    return url

Serverless LambdaSharp serverless.yml

# serverless.yml
service: s3upload

provider:
  name: aws
  runtime: python3.7
  versionFunctions: false
  memorySize: 128
  timeout: 30

  # you can add statements to the Lambda function's IAM Role here
  iamRoleStatements: 
    - Effect: "Allow"
      Action:
        - s3:PutObject
        - s3:GetObject
      Resource:
        - Fn::Join: ["/", [Fn::GetAtt: [FilesBucket, Arn], "*"]]
        - Fn::GetAtt: [FilesBucket, Arn]

# Package information
package:
  artifact: package.zip

functions:
  s3upload-function:
    handler: lambda_function.lambda_handler
    environment:
      FILES_BUCKET:
        Ref: FilesBucket
    events: 
      # THIS LAMBDA FUNCTION WILL BE TRIGGERED EVERY 10 MINUTES
      # CHECK OUT THE SERVERLESS DOCS FOR ALTERNATIVE WAYS TO 
      # TRIGGER THE FUNCTION
       - schedule:
           rate: rate(10 minutes)

# you can add CloudFormation resource templates here
resources:
  Resources:
    FilesBucket:
      Type: AWS::S3::Bucket
      Properties:
        PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true

#!/usr/bin/env bash
# deploy.sh
mkdir package
pip install -r requirements.txt --target=./package
cp lambda_function.py package/
$(cd package; zip -r ../package.zip .)
serverless deploy --verbose