S3

S3 contains a variety of AWS S3 related operations commonly used by the DAP team

nesta_ds_utils.loading_saving.S3.upload_obj(obj: any, bucket: str, path_to: str, kwargs_boto: dict = {}, kwargs_writing: dict = {})[source]

Uploads data from memory to S3 location.

Parameters:

  • obj (any) – Data to upload.

  • bucket (str) – Bucket’s name.

  • path_to (str) – Path location to save data.

  • kwargs_boto (dict, optional) – Dictionary of kwargs for boto3 function ‘upload_fileobj’.

  • kwargs_writing (dict, optional) – Dictionary of kwargs for writing data.

nesta_ds_utils.loading_saving.S3.download_obj(bucket: str, path_from: str, download_as: Optional[str] = None, kwargs_boto: dict = {}, kwargs_reading: dict = {}) → any[source]

Download data to memory from S3 location.

Parameters:

  • bucket (str) – Bucket’s name.

  • path_from (str) – Path to data in S3.

  • download_as (str, optional) – Type of object downloading. Choose between

  • ('dataframe' –

  • 'geodf' –

  • 'dict' –

  • 'list' –

  • 'str' –

  • files'. ('np.array'). Not needed for 'pkl) –

  • kwargs_boto (dict, optional) – Dictionary of kwargs for boto3 function ‘download_fileobj’.

  • kwargs_reading (dict, optional) – Dictionary of kwargs for reading data.

Returns:

Downloaded data.

Return type:

any

nesta_ds_utils.loading_saving.S3.upload_file(path_from: str, bucket: str, path_to: str, **kwargs)[source]

Upload local file from disk to S3 location.

Parameters:

  • path_from (str) – Path to local file.

  • bucket (str) – Bucket’s name.

  • path_to (str) – Destination path in S3.

nesta_ds_utils.loading_saving.S3.download_file(path_from: str, bucket: str, path_to: str, **kwargs)[source]

Download data from S3 to local file on disk.

Parameters:

  • path_from (str) – Path to data in S3

  • bucket (str) – Buket’s name

  • path_to (str) – Destination path to disk.