How to specify credentials when connecting to boto3 S3?

python amazon-web-services amazon-s3 boto3

On boto I used to specify my credentials when connecting to S3 in such a way:

import boto
from boto.s3.connection import Key, S3Connection
S3 = S3Connection( settings.AWS_SERVER_PUBLIC_KEY, settings.AWS_SERVER_SECRET_KEY )

I could then use S3 to perform my operations (in my case deleting an object from a bucket).

With boto3 all the examples I found are such:

import boto3
S3 = boto3.resource( 's3' )
S3.Object( bucket_name, key_name ).delete()

I couldn't specify my credentials and thus all attempts fail with InvalidAccessKeyId error.

How can I specify credentials with boto3?

This answer might help you: stackoverflow.com/a/36913771/2681632

See the "Configuring Credentials" section in the official documentation: boto3.readthedocs.io/en/latest/guide/configuration.html

I find it super strange to call this 'AWS_SERVER_PUBLIC_KEY'. The name is 'access key id' and has nothing to do with the public part of a keypair. Also an access to a service like s3 should not be confused with a server(host) access. And i recommend to not let this key id becoming public (even if it's useless alone)

Alasdair

You can create a session:

import boto3
session = boto3.Session(
    aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
    aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
)

Then use that session to get an S3 resource:

s3 = session.resource('s3')

works, I will take it as the answer. Why on earth don't they document this as the obvious way to do it?!!

As mentioned in a comment above, this is in fact in the documentation.

@Moot I was initially going to say I couldn't find this in the docs but under credentials in the 'warning' section it notes that the token is optional. I don't really get why you would ever bother with that.

@JimmyJames this is getting off topic, but you can use AWS STS to generate temporary credentials (e.g. valid for one hour). In that case, the session token is required, it won't work if you omit it.

@JimmyJames the use case for STS is that you start with aws_access_key_id and aws_secret_access_key which have limited permissions. They don't allow you access S3, but they do allow you to assume a role which can access S3. You make the AWS STS call to assume the role, which returns an new aws_access_key_id, aws_secret_access_key and aws_session_token combination (the key and access key are different from the originals). You then use these credentials to create a new session to access S3.

Alasdair

You can get a client with new session directly like below.

 s3_client = boto3.client('s3', 
                      aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY, 
                      aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY, 
                      region_name=REGION_NAME
                      )

This works for getting an s3 client, but the OP wanted an s3 resource instead.

I agree with @Alasdair. The docs don't show how to do anything with client, and neither do you, so I don't see how this answer is relevant.

I tried this but it gives me "Unable to locate credentials" error..I had previously removed ~/.aws folder to test this as i know that boto will by default look for creds there...

This is the right answer and the only method that works as today. I don't know what you guys are talking about this not being useful. You can do ANYTHING using the client and there's extensive documentation for EVERY AWS service. Just take a look for S3: boto3.amazonaws.com/v1/documentation/api/latest/reference/…

DataDecay

This is older but placing this here for my reference too. boto3.resource is just implementing the default Session, you can pass through boto3.resource session details.

Help on function resource in module boto3:

resource(*args, **kwargs)
    Create a resource service client by name using the default session.

    See :py:meth:`boto3.session.Session.resource`.

https://github.com/boto/boto3/blob/86392b5ca26da57ce6a776365a52d3cab8487d60/boto3/session.py#L265

you can see that it just takes the same arguments as Boto3.Session

import boto3
S3 = boto3.resource('s3', region_name='us-west-2', aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY, aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY)
S3.Object( bucket_name, key_name ).delete()

x85ms16

I'd like expand on @JustAGuy's answer. The method I prefer is to use AWS CLI to create a config file. The reason is, with the config file, the CLI or the SDK will automatically look for credentials in the ~/.aws folder. And the good thing is that AWS CLI is written in python.

You can get cli from pypi if you don't have it already. Here are the steps to get cli set up from terminal

$> pip install awscli  #can add user flag 
$> aws configure
AWS Access Key ID [****************ABCD]:[enter your key here]
AWS Secret Access Key [****************xyz]:[enter your secret key here]
Default region name [us-west-2]:[enter your region here]
Default output format [None]:

After this you can access boto and any of the api without having to specify keys (unless you want to use a different credentials).

You can also specify the column you want to fill : - aws configure set aws_access_key_id [****************ABCD] - aws configure set aws_secret_access_key [****************xyz]

JustAGuy

There are numerous ways to store credentials while still using boto3.resource(). I'm using the AWS CLI method myself. It works perfectly.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html?fbclid=IwAR2LlrS4O2gYH6xAF4QDVIH2Q2tzfF_VZ6loM3XfXsPAOR4qA-pX_qAILys

Bernard

If you rely on your .aws/credentials to store id and key for a user, it will be picked up automatically.

For instance

session = boto3.Session(profile_name='dev')
s3 = session.resource('s3')

This will pick up the dev profile (user) if your credentials file contains the following:

[dev]
aws_access_key_id = AAABBBCCCDDDEEEFFFGG
aws_secret_access_key = FooFooFoo
region=op-southeast-2

Den Roman

you can set default aws env variables for secret and access keys - that way you dont need to change default client creation code - though it is better to pass it as a parameter if you have non-default creds

How to specify credentials when connecting to boto3 S3?

Follow WeChat

Want to stay one step ahead of the latest teleworks?

相似问题

Platform

Support

Contact US