Adapter
Adapter for Integration to Amazon Web Services S3
Overview
This adapter is used to integrate the Itential Automation Platform (IAP) with the Awss3 System. The API that was used to build the adapter for Awss3 is usually available in the report directory of this adapter. The adapter utilizes the Awss3 API to provide the integrations that are deemed pertinent to IAP. The ReadMe file is intended to provide information on this adapter it is generated from various other Markdown files.
Details
The AWS S3 adapter from Itential is used to integrate the Itential Automation Platform (IAP) with AWS S3. With this adapter you have the ability to perform operations such as:
- Backup and restore data.
- Configure and Manage access constrols.
- Configure compliance requirements.
For further technical details on how to install and use this adapter, please click the Technical Documentation tab.
Amazon AWS S3
Table of Contents
Specific Adapter Information
Authentication
This document will go through the steps for authenticating the AWS S3 adapter with AWS Signature 4 Authentication. Properly configuring the properties for an adapter in IAP is critical for getting the adapter online. You can read more about adapter authentication HERE.
AWS Authentication
The AWS S3 adapter requires AWS Authentication therefore the auth_method should be set to aws_authentication. The adapter utilizes AWS signature 4 authentication. There are three mechanisms for doing this.
The first way is using a "service" account and its AWS keys to authenticate as that account. In this case, you will get the aws_access_key, aws_secret_key, and aws_session_token from AWS and configure them into the adapter service instance as shown below.
The second way is using AWS STS. this still requires a "service" account and its AWS keys to authenticate as that account. In this case, you will get the aws_access_key, aws_secret_key, and aws_session_token from AWS and configure them into the adapter service instance as shown below. In addition, you will provide STS paramaters in the workflow tasks that tell the adapter the role you want used on the particular call.
The third authentication method is to use an IAM role. With this method, you do not need any authentication keys as the adapter will utilize an "internal" AWS call to get the things that it needs for authentication. Since the adapter needs to make the call to this "internal" AWS IP address, the IAP server needs to be where it has access to that address or you will not be able to use this method.
If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
AWS Signature 4 Service Account Authentication
The AWS S3 adapter requires AWS Signature 4 Authentication. If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.
STEPS
- Ensure you have access to a AWS S3 server and that it is running
- Follow the steps in the README.md to import the adapter into IAP if you have not already done so
- Use the properties below for the
properties.authenticationfield"authentication": { "auth_method": "aws_authentication", "aws_access_key": "aws_access_key", "aws_secret_key": "aws_secret_key", "aws_session_token": "aws_session_token" }you can leave all of the other properties in the authentication section, they will not be used for AWS S3 authentication.
- Restart the adapter. If your properties were set correctly, the adapter should go online.
AWS Security Token Service
The AWS S3 adapter also supports AWS Security Token Service (STS) Authentication. For using this authentication, you need to use the calls in the Adapter that have the STSRole suffix on them and pass the STS information into the method. You will still need to provide the relevant aws_secret_key and aws_access_key as described above. Below is an example of the data required in the sts tasks:
{
"RoleArn": "arn:aws:iam::1234567:role/my_role",
"RoleSessionName": "mySession"
}The AWS STS Authentication goes to the AWS STS Service endpoint in order to validate that the primary "service" account the adapter has authenticated with has the permission to assume the role. This call is made to sts.amazonaws.com or a regional sts sevice (e.g. sts.us-east-1.amazonaws.com). By default traffic to these endpoints will go out through the Internet. In the case where you would prefer these route through your network, it is possible to change the STS config for the adapter. The proxy field should point to the AWS loadbalancer or a proxy server that forwards to AWS STS. In Itential Cloud, this can be NAT'd to your network. In addition to this, you may need to set the endpoint in order to have the STS SSL certificate validated successfully. By default the adapter will use sts regional servers. If the loadbalancer and proxy are set up for that you should be fine. If however, they point to the global STS service (sts.amazonaws.com) You will need to set the global as the endpoint or the STS certificate will be rejected due to the hosts not matching.
"authentication": {
"aws_sts": {
"endpoint": "<sts certificate endpoint>",
"proxy": "<proxy/loadbalancer ip>",
}
}AWS IAM Role
The AWS S3 adapter also supports AWS IAM Role Authentication. For using this authentication, you need to use the calls in the Adapter that have the STSRole suffix on them and provide the role's ARN in the RoleName variable. In addition to passing the IAM Role in the task, it is possible to set an IAM Role in the Service Instance Configuration by using the aws_iam_role property in the authentication section and providing the role's ARN.
"authentication": {
"auth_method": "aws_authentication",
"aws_iam_role": "role_arn"
}AMAZON STEPS FOR IAM ROLE
Increase number of hops if running IAP inside of docker on an AWS instance
aws sso login --profile aws-bota-1
<export aws keys for CLI access>
Amazon ec2 modify-instance-metadata-options --instance-id i-0e150236026b7c45d --http-put-response-hop-limit 3 --http-endpoint enabled --region us-east-1Create a new role and attach to it policies:
- go to your S3 instance, select it
- Actions->Security->Modify IAM Role
- Click 'Create New IAM Role'
- Create a role:
Trusted entity type: AWS service Use Case: S3
Add needed S3 policies to the role
Save the role
Go back to S3 and Actions->Security->Modify IAM Role, associate newly created role with your S3 instance
Troubleshooting
- Make sure you copied over the correct access key, secret key and session token.
- Turn on debug level logs for the adapter in IAP Admin Essentials.
- Turn on auth_logging for the adapter in IAP Admin Essentials (adapter properties).
- Investigate the logs - in particular:
- The FULL REQUEST log to make sure the proper headers are being sent with the request.
- The FULL BODY log to make sure the payload is accurate.
- The CALL RETURN log to see what the other system is telling us.
- Credentials should be masked by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
- Remember when you are done to turn auth_logging off as you do not want to log credentials.
- For IAM, you can run this on the IAP server to verify you are getting to the "internal" AWS Server
TOKEN=`curl -v -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -v -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/<rolename>Sample Properties
Sample Properties can be used to help you configure the adapter in the Itential Automation Platform. You will need to update connectivity information such as the host, port, protocol and credentials.
"properties": {
"host": "s3.amazonaws.com",
"region": "us-east-1",
"port": 443,
"choosepath": "",
"base_path": "/",
"version": "",
"cache_location": "none",
"encode_pathvars": true,
"encode_queryvars": true,
"save_metric": false,
"stub": true,
"protocol": "https",
"service": "s3",
"authentication": {
"auth_method": "aws_authentication",
"username": "username",
"password": "password",
"token": "token",
"token_timeout": 180000,
"token_cache": "local",
"invalid_token_error": 401,
"auth_field": "header.headers.Cookie",
"auth_field_format": "Token {token}",
"auth_logging": false,
"client_id": "",
"client_secret": "",
"grant_type": "",
"sensitive": [],
"sso": {
"protocol": "",
"host": "",
"port": 0
},
"multiStepAuthCalls": [
{
"name": "",
"requestFields": {},
"responseFields": {},
"successfullResponseCode": 200
}
],
"aws_access_key": "aws_access_key",
"aws_secret_key": "aws_secret_key",
"aws_session_token": "aws_session_token",
"aws_iam_role": "",
"aws_sts": {
"sslEnable": true,
"endpoint": "",
"proxy": "",
"proxyagent": ""
}
},
"healthcheck": {
"type": "none",
"frequency": 60000,
"query_object": {},
"addlHeaders": {}
},
"throttle": {
"throttle_enabled": false,
"number_pronghorns": 1,
"sync_async": "sync",
"max_in_queue": 1000,
"concurrent_max": 1,
"expire_timeout": 0,
"avg_runtime": 200,
"priorities": [
{
"value": 0,
"percent": 100
}
]
},
"request": {
"number_redirects": 0,
"number_retries": 3,
"limit_retry_error": 0,
"failover_codes": [],
"attempt_timeout": 5000,
"global_request": {
"payload": {},
"uriOptions": {},
"addlHeaders": {},
"authData": {}
},
"healthcheck_on_timeout": true,
"return_raw": false,
"archiving": false,
"return_request": false
},
"proxy": {
"enabled": false,
"host": "",
"port": 1,
"protocol": "http",
"username": "",
"password": ""
},
"ssl": {
"ecdhCurve": "",
"enabled": false,
"accept_invalid_cert": true,
"ca_file": "",
"key_file": "",
"cert_file": "",
"secure_protocol": "",
"ciphers": ""
},
"mongo": {
"host": "",
"port": 0,
"database": "",
"username": "",
"password": "",
"replSet": "",
"db_ssl": {
"enabled": false,
"accept_invalid_cert": false,
"ca_file": "",
"key_file": "",
"cert_file": ""
}
},
"devicebroker": {
"enabled": false,
"getDevice": [
{
"path": "/not/mapped",
"method": "GET",
"query": {},
"body": {},
"headers": {},
"handleFailure": "ignore",
"requestFields": {
"insample": "{port}"
},
"responseDatakey": "",
"responseFields": {
"name": "{this}{||}{that}",
"ostype": "{osfield}",
"ostypePrefix": "meraki-",
"port": "{port}",
"ipaddress": "{ip_addr}",
"serial": "{serial}"
}
}
],
"getDevicesFiltered": [
{
"path": "/not/mapped",
"method": "GET",
"pagination": {
"offsetVar": "",
"limitVar": "",
"incrementBy": "limit",
"requestLocation": "query"
},
"query": {},
"body": {},
"headers": {},
"handleFailure": "ignore",
"requestFields": {},
"responseDatakey": "",
"responseFields": {
"name": "{this}{||}{that}",
"ostype": "{osfield}",
"ostypePrefix": "meraki-",
"port": "{port}",
"ipaddress": "{ip_addr}",
"serial": "{serial}",
"id": "{myid}"
}
}
],
"isAlive": [
{
"path": "/not/mapped/{devID}",
"method": "GET",
"query": {},
"body": {},
"headers": {},
"handleFailure": "ignore",
"requestFields": {
"devID": "{id}"
},
"responseDatakey": "",
"responseFields": {
"status": "return2xx",
"statusValue": "AD.200"
}
}
],
"getConfig": [
{
"path": "/not/mapped/{devID}",
"method": "GET",
"query": {},
"body": {},
"headers": {},
"handleFailure": "ignore",
"requestFields": {
"devID": "{id}"
},
"responseDatakey": "",
"responseFields": {}
}
],
"getCount": [
{
"path": "/not/mapped",
"method": "GET",
"query": {},
"body": {},
"headers": {},
"handleFailure": "ignore",
"requestFields": {},
"responseDatakey": "",
"responseFields": {}
}
]
},
"cache": {
"enabled": false,
"entities": [
{
"entityType": "device",
"frequency": 3600,
"flushOnFail": false,
"limit": 10000,
"retryAttempts": 5,
"sort": true,
"populate": [
{
"path": "/not/mapped",
"method": "GET",
"pagination": {
"offsetVar": "",
"limitVar": "",
"incrementBy": "limit",
"requestLocation": "query"
},
"query": {},
"body": {},
"headers": {},
"handleFailure": "ignore",
"requestFields": {},
"responseDatakey": "",
"responseFields": {
"name": "{this}{||}{that}",
"ostype": "{osfield}",
"ostypePrefix": "meraki-",
"port": "{port}",
"ipaddress": "{ip_addr}",
"serial": "{serial}",
"id": "{myid}"
}
}
],
"cachedTasks": [
{
"name": "",
"filterField": "",
"filterLoc": ""
}
]
}
]
}
}