Interface to Amazon S3

Airflow provides an interface to Amazon S3 by using the Providers.

See Provider examples of Amazon S3 at <airflow_home>/build/env/lib/python3.9/site-packages/airflow/providers/ezmeral/s3/example_dags/.

Hooks

EzS3Hook
Python path: airflow.providers.ezmeral.s3.hooks.s3
Description: Interacts with Amazon S3 using Boto3 library.

Operators

EzS3CreateBucketOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_bucket
Description: Creates an S3 bucket.
EzS3DeleteBucketOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_bucket
Description: Deletes an S3 bucket.
EzS3GetBucketTaggingOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging
Description: Gets tagging from an S3 bucket.
EzS3PutBucketTaggingOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging
Description: Puts tagging for an S3 bucket.
EzS3DeleteBucketTaggingOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging
Description: Deletes tagging from an S3 bucket.
EzS3CopyObjectOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_copy_object
Description: Creates a copy of the stored S3 object.
EzS3DeleteObjectsOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_delete_objects
Description: Deletes single or multiple objects from a bucket using a single HTTP request.
EzS3ListOperator
Python path: airflow.providers.ezmeral.s3.operators.s3_list
Description: List all objects from the bucket with the given string prefix in name.

Sensors

EzS3KeySensor
Python path: airflow.providers.ezmeral.s3.sensors.s3_key
Description: Waits for a key to show up in a S3 bucket. S3 is a key-value store and key uniquely identifies the object in the bucket. S3 does not support folders and the path is just a key of resource.
EzS3KeyUnchangedSensor
Python path: airflow.providers.ezmeral.s3.sensors.s3_keys_unchanged
Description: Checks for changes in the number of objects at prefix in S3 bucket. This sensor returns True if the inactivity period has passed with no increase in the number of objects. This sensor might give an error when you use it in reschedule mode. Between rescheduled invocations, S3 bucket loses the state of the listed objects.

Transfers

EzGCSToS3Operator
Python path: airflow.providers.ezmeral.s3.transfers.gcs_to_s3
Description: Synchronizes a Google Cloud Storage bucket with an Amazon S3 bucket.
EzLocalFilesystemToS3Operator
Python path: airflow.providers.ezmeral.s3.transfers.local_to_s3
Description: Uploads a file from a local filesystem to S3.
EzMongoToS3Operator
Python path: airflow.providers.ezmeral.s3.transfers.mongo_to_s3
Description: Moves data from MongoDB using PyMongo to Amazon S3 using Boto.
EzSQLToS3Operator
Python path: airflow.providers.ezmeral.s3.transfers.sql_to_s3.
Description: Saves data from a specific SQL query into a file in S3.
EzS3ToFTPOperator
Python path: airflow.providers.ezmeral.s3.transfers.s3_to_ftp
Description: Enables the transfer of files from S3 to a FTP server.
EzS3ToSFTPOperator
Python path: airflow.providers.ezmeral.s3.transfers.s3_to_sftp
Description: Enables the transfer of files from S3 to a SFTP server.
EzSFTPToS3Operator
Python path: airflow.providers.ezmeral.s3.transfers.sftp_to_s3
Description: Enables the transfer of files from a SFTP server to S3.