now.executor.preprocessor.s3_download module#
- now.executor.preprocessor.s3_download.maybe_download_from_s3(docs, tmpdir, user_input, max_workers)[source]#
Downloads file to local temporary dictionary, saves S3 URI to tags[‘uri’] and modifies uri attribute of document to local path in-place.
- Parameters
doc – document containing URI pointing to the location on S3 bucket
tmpdir (
TemporaryDirectory
) – temporary directory in which files will be saveduser_input – User iput which contain aws credentials
max_workers – number of threads to create in the threadpool executor to make execution faster
- now.executor.preprocessor.s3_download.convert_fn(d, tmpdir, aws_access_key_id, aws_secret_access_key, aws_region_name)[source]#
Downloads files and tags from S3 bucket and updates the content uri and the tags uri to the local path
- Return type
Document