Step 1: Create an amazon.com account via Amazon.com.
Step 2: Create an aws.amazon.com account. You need to provide credit/debit card details even if you want to use the free tier for up to 12 months. It is advisable to use a credit/debit card with low limits (E.g. getting a debit card from a post office with say $20 limit). Also, be aware of the AWS pricing. Read up on https://aws.amazon.com/free/ and https://aws.amazon.com/pricing/services/.
If you face issues signing-up related to mobile phone number verification, etc raise a support ticket with “aws.amazon.com”.
Step 3: Once you are logged into the AWS console, you can set up the basic security required, especially for any programmatic access from say a Spark job or a RESTful Web Service.
AWS Security
1) You use an email address and password to sign in to secure pages on AWS, such as the AWS Management Console, AWS Forums, and AWS Support.
AWS Console -> Security, Identity & Compliance -> IAM (Identity & Access Management) -> Password
2) You use access keys to make programmatic calls to AWS API operations.
How do you get the API access keys?
Step 4: Go to AWS Console Services and then
AWS Console -> Security, Identity & Compliance -> IAM (Identity & Access Management) -> “Manage Security Credentials” -> Access keys (access key ID and secret access key)
Important: When you create an access key, AWS gives you an opportunity to view and download the secret access key only once. If you don’t download it or if you lose it, you can delete the access key and then create a new one.
Where will I use an access key?
You need it to access any AWS APIs like RESTful Service calls, accessing the S3 storage, etc. For example, if you are accessing the Amazon storage S3 from a Spark job you may have to enter the access & secret keys as described below.
Option 1: In the Hadoop’s “core-site.xml” file enter the access & secret keys.
1 2 3 4 5 6 7 8 9 10 |
<property> <name>fs.s3a.access.key</name> <value>...</value> </property> <property> <name>fs.s3a.secret.key</name> <value>...</value> </property> |
Option 2: Specify the credentials at run time.
1 2 3 4 |
sc.hadoopConfiguration.set("fs.s3a.access.key", "...") sc.hadoopConfiguration.set("fs.s3a.secret.key", "...") |
Note: Any issues regarding creating accounts create a case with AWS support.