now.executor.preprocessor.s3_download module

now.executor.preprocessor.s3_download.maybe_download_from_s3(docs, tmpdir, user_input, max_workers)

Downloads file to local temporary dictionary, saves S3 URI to tags[‘uri’] and modifies uri attribute of document to local path in-place.

Parameters

• docs (DocumentArray) – documents containing URI pointing to the location on S3 bucket

• tmpdir (TemporaryDirectory) – temporary directory in which files will be saved

• user_input – User iput which contain aws credentials

• max_workers – number of threads to create in the threadpool executor to make execution faster

now.executor.preprocessor.s3_download.convert_s3_to_local_uri(d, tmpdir, aws_access_key_id, aws_secret_access_key, aws_region_name)

Downloads files and tags from S3 bucket and updates the content uri and the tags uri to the local path

Return type

Document

now.executor.preprocessor.s3_download.get_bucket(uri, aws_access_key_id, aws_secret_access_key, region_name)

now.executor.preprocessor.s3_download.download_from_bucket(tmpdir, uri, bucket)

now.executor.preprocessor.s3_download.get_dict_value_for_flattened_key(d, keys)

now.executor.preprocessor.s3_download.get_local_path(tmpdir, path_s3)

now.executor.preprocessor.s3_download.update_tags(d, aws_access_key_id, aws_secret_access_key, region_name)