Amazon S3

This is the implementation of the S3 data handler for MindsDB.

Amazon Simple Storage Service, or Amazon S3, is an object storage service that offers industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can use Amazon S3 to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides management features so that you can optimize, organize, and configure access to your data to meet your specific business, organizational, and compliance requirements.

Implementation

This handler is implemented using boto3, the AWS SDK for Python.

The required arguments to establish a connection are as follows:

• aws_access_key_id is the AWS access key that identifies the user or IAM role.
• aws_secret_access_key is the AWS secret access key that identifies the user or IAM role.
• region_name is the AWS region.
• bucket is the name of the S3 bucket.
• key is the key of the object to be queried.
• input_serialization is the format of the data in the object that is to be queried.

If you installed MindsDB locally via pip, you need to install all handler dependencies manually. To do so, go to the handler’s folder (mindsdb/integrations/handlers/s3_handler) and run this command: pip install -r requirements.txt .

Usage

In order to make use of this handler and connect to an object in the S3 bucket from MindsDB, the following syntax can be used:

CREATE DATABASE s3_datasource
WITH
    engine = 's3',
    parameters = {
      "aws_access_key_id": "aws_access_key_id_value",
      "aws_secret_access_key": "aws_secret_access_key_value",
      "region_name": "region_name",
      "bucket": "bucket_name",
      "key": "fine_name.csv",
      "input_serialization": "{'CSV': {'FileHeaderInfo': 'NONE'}}"
    };

You can use this established connection to query your object as follows:

SELECT *
FROM s3_datasource.S3Object;

Queries to objects in the S3 bucket are issued using S3 Select.

This feature is used in boto3 as described here.

The required format of the InputSerialization parameter, which translates to the input_serialization parameter of the handler, is of special importance, as it describes how to specify the format of the data in the object that is to be queried.

At the moment, S3 Select does not allow multiple files to be queried. Therefore, queries can be issued only to the object that is passed in the key parameter. This object should always be referred to as an S3Object when writing queries as shown in the example above.