now.common.detect_schema module#

now.common.detect_schema.get_field_type(field_value)[source]#
now.common.detect_schema.get_first_file_in_folder_structure_s3(bucket, folder_prefix, dataset_path)[source]#
now.common.detect_schema.set_field_names_from_docarray(user_input, **kwargs)[source]#

Get the schema from a DocArray

Parameters

user_input (UserInput) – UserInput object

Makes a request to hubble API and downloads the first 10 documents from the document array and uses the first document to get the schema and sets field_names in user_input

now.common.detect_schema.set_field_names_from_s3_bucket(user_input, **kwargs)[source]#

Get the schema from a S3 bucket

Parameters

user_input (UserInput) – UserInput object

checks if the bucket exists and the format of the folder structure is correct, if yes then downloads the first folder and sets its content as fields_dict in user_input

now.common.detect_schema.set_field_names_from_local_folder(user_input, **kwargs)[source]#

Get the schema from a local folder

Parameters

user_input (UserInput) – UserInput object

checks if the folder exists and the format of the folder structure is correct, if yes set the content of the first folder as field_names in user_input

now.common.detect_schema.set_field_names_elasticsearch(user_input, **kwargs)[source]#

Get the schema from an Elasticsearch instance

Parameters

user_input (UserInput) – UserInput object

checks if the Elasticsearch instance exists and grabs the first document from the index, the first document is then used to create modalities dicts for index and filter fields

now.common.detect_schema.get_s3_bucket_and_folder_prefix(user_input)[source]#

Gets the s3 bucket and folder prefix from the user input.

Parameters

user_input (UserInput) – The user input

Returns

The s3 bucket and folder prefix