Airflow provides an interface to Amazon S3 by using the Providers.
See Provider examples of Amazon S3 at
<airflow_home>/build/env/lib/python3.9/site-packages/airflow/providers/ezmeral/s3/example_dags/.
Hooks
- EzS3Hook
- Python path: airflow.providers.ezmeral.s3.hooks.s3
- Description: Interacts with Amazon S3 using Boto3 library.
Operators
- EzS3CreateBucketOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_bucket
- Description: Creates an S3 bucket.
- EzS3DeleteBucketOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_bucket
- Description: Deletes an S3 bucket.
- EzS3GetBucketTaggingOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging
- Description: Gets tagging from an S3 bucket.
- EzS3PutBucketTaggingOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging
- Description: Puts tagging for an S3 bucket.
- EzS3DeleteBucketTaggingOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_bucket_tagging
- Description: Deletes tagging from an S3 bucket.
- EzS3CopyObjectOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_copy_object
- Description: Creates a copy of the stored S3 object.
- EzS3DeleteObjectsOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_delete_objects
- Description: Deletes single or multiple objects from a bucket using a single
HTTP request.
- EzS3ListOperator
- Python path: airflow.providers.ezmeral.s3.operators.s3_list
- Description: List all objects from the bucket with the given string prefix in
name.
Sensors
- EzS3KeySensor
- Python path: airflow.providers.ezmeral.s3.sensors.s3_key
- Description: Waits for a key to show up in a S3 bucket. S3 is a key-value store
and key uniquely identifies the object in the bucket. S3 does not support folders and
the path is just a key of resource.
- EzS3KeySizeSensor
- Python path: airflow.providers.ezmeral.s3.sensors.s3_key
- Description: Waits for a key whose size is larger than the defined size to show
up in a S3 bucket. S3 is a key-value store and key uniquely identifies the object in the
bucket. S3 does not support folders and the path is just a key of resource.
- EzS3KeyUnchangedSensor
- Python path: airflow.providers.ezmeral.s3.sensors.s3_keys_unchanged
- Description: Checks for changes in the number of objects at prefix in S3
bucket. This sensor returns
True if the inactivity period has passed
with no increase in the number of objects. This sensor might give an error when you use
it in reschedule mode. Between rescheduled invocations, S3 bucket loses the state of the
listed objects.
- EzS3PrefixSensor
- Python path: airflow.providers.ezmeral.s3.sensors.s3_prefix
- Description: Waits for a prefix to be available. A prefix is the first part of
a key which enables checking of constructs similar to glob airfl* or SQL LIKE 'airfl%'.
You can specify a delimiter to indicate the hierarchy or key which stops the match at
that delimiter.
Transfers
- EzGCSToS3Operator
- Python path: airflow.providers.ezmeral.s3.transfers.gcs_to_s3
- Description: Synchronizes a Google Cloud Storage bucket with an Amazon S3
bucket.
- EzLocalFilesystemToS3Operator
- Python path: airflow.providers.ezmeral.s3.transfers.local_to_s3
- Description: Uploads a file from a local filesystem to S3.
- EzMongoToS3Operator
- Python path: airflow.providers.ezmeral.s3.transfers.mongo_to_s3
- Description: Moves data from MongoDB using PyMongo to Amazon S3 using
Boto.
- EzMySQLToS3Operator
- Python path: airflow.providers.ezmeral.s3.transfers.mysql_to_s3
- Description: Saves data from a specific MySQL query into a file in S3.
- EzS3ToFTPOperator
- Python path: airflow.providers.ezmeral.s3.transfers.s3_to_ftp
- Description: Enables the transfer of files from S3 to a FTP server.
- EzS3ToSFTPOperator
- Python path: airflow.providers.ezmeral.s3.transfers.s3_to_sftp
- Description: Enables the transfer of files from S3 to a SFTP server.
- EzSFTPToS3Operator
- Python path: airflow.providers.ezmeral.s3.transfers.sftp_to_s3
- Description: Enables the transfer of files from a SFTP server to S3.