Amazon AWS S3

Table of Contents

• Specific Adapter Information
  • Authentication
  • Sample Properties
  • Swagger
• Generic Adapter Information

Specific Adapter Information

Authentication

This document will go through the steps for authenticating the AWS S3 adapter with AWS Signature 4 Authentication. Properly configuring the properties for an adapter in IAP is critical for getting the adapter online. You can read more about adapter authentication HERE.

AWS Authentication

The AWS S3 adapter requires AWS Authentication therefore the auth_method should be set to aws_authentication. The adapter utilizes AWS signature 4 authentication. There are three mechanisms for doing this. There is a brief description in this section and more information in the specific section for each authentication.

The first way is using a "service" account and its AWS keys to authenticate as that account. In this case, you will get the aws_access_key, aws_secret_key, and aws_session_token from AWS and configure them into the adapter service instance as shown below.

The second way is using AWS STS. this still requires a "service" account and its AWS keys to authenticate as that account. In this case, you will get the aws_access_key, aws_secret_key, and aws_session_token from AWS and configure them into the adapter service instance as shown below. In addition, you will provide STS paramaters in the workflow tasks that tell the adapter the role you want used on the particular call.

The third authentication method is to use an IAM role. With this method, you do not need any authentication keys as the adapter will utilize an "internal" AWS call to get the things that it needs for authentication. Since the adapter needs to make the call to this "internal" AWS IP address, the IAP server needs to be where it has access to that address or you will not be able to use this method.

If you change authentication methods, you should change this section accordingly and merge it back into the adapter repository.

AWS Signature 4 Service Account Authentication

This can also be referred to as standard credential signing using an Adapter Role or Pod Role.

Provide access key and secret key in adapter service instance configuration (Adapter Role). Those credentials are used to sign all calls. If no access key and/or secret key are provided, it will use AWS Environment variables (which in Saas is the Pod Role) to sign all calls. If this role has no permission to make the call, the call will fail with authentication issues.

STEPS

1. Ensure you have access to a AWS S3 server and that it is running
2. Follow the steps in the README.md to import the adapter into IAP if you have not already done so
3. Use the properties below for the properties.authentication field

   "authentication": {
   "auth_method": "aws_authentication",
   "aws_access_key": "aws_access_key",
   "aws_secret_key": "aws_secret_key",
   "aws_session_token": "aws_session_token"
   }

   you can leave all of the other properties in the authentication section, they will not be used for AWS S3 authentication.

4. Restart the adapter. If your properties were set correctly, the adapter should go online.

AWS Security Token Service

The AWS S3 adapter also supports AWS Security Token Service (STS) Authentication. This can also be referred to as standard credential but then STS assume role based on STS Params on Task. Adapter Role => Task Role(s) or Pod Role => Task Role(s).

Provide access key and secret key in adapter service instance configuration (Adapter Role). Will use these credentials on the assume role request to assume a different role defined in the STSParams on the task (Task Role). The Task Role will then be used to sign the call to AWS. The Adapter Role has to have the right to assume the Task Role or the call will fail. If no access key and/or secret key are provided, it will use AWS Environment variables (which is Saas is the Pod Role) to attempt the assume a different role defined in the STSParams on the task (Task Role). The Task Role will then be used to sign the call to AWS. The Pod Role has to have the right to assume the Task Role or the call will fail.

For using this authentication, you need to use the calls in the Adapter that have the STSRole suffix on them and pass the STS information into the method. You will still need to provide the relevant aws_secret_key and aws_access_key as described above. Below is an example of the data required in the sts tasks:

{
  "RoleArn": "arn:aws:iam::1234567:role/my_role",
  "RoleSessionName": "mySession"
}

The AWS STS Authentication goes to the AWS STS Service endpoint in order to validate that the primary "service" account the adapter has authenticated with has the permission to assume the role. This call is made to sts.amazonaws.com or a regional sts sevice (e.g. sts.us-east-1.amazonaws.com). By default traffic to these endpoints will go out through the Internet. In the case where you would prefer these route through your network, it is possible to change the STS config for the adapter.

The proxy field should point to the AWS loadbalancer or a proxy server that forwards to AWS STS. In Itential Cloud, this can be NAT'd to your network. In addition to this, you may need to set the endpoint in order to have the STS SSL certificate validated successfully. By default the adapter will use sts regional servers. If the loadbalancer and proxy are set up for that you should be fine. If however, they point to the global STS service (sts.amazonaws.com) You will need to set the global as the endpoint or the STS certificate will be rejected due to the hosts not matching.

Region can be important as it is the region in which the STS assume role request will be processed. Each AWS partition may have one primary region for STS. In our tests, we have found that for the standard partition the STS region should be set to us-east-t but this is configurable should your primary region be different or you are working in a different AWS partition.

If doing cross organizational role assumption it is recommended to have an external id that is agreed upon for further security. THis external id can be provided in the aws_sts properties (global) or defined in the STS Params object on the task (dynamic).

"authentication": {
  "aws_sts": {
    "region": "us-east-1",
    "endpoint": "<sts certificate endpoint>",
    "proxy": "<proxy/loadbalancer ip>",
    "externalId": "<sts external id>"
  }
}

AWS IAM Role

The AWS S3 adapter also supports AWS IAM Role Authentication. For using this authentication, you need to use the calls in the Adapter that have the STSRole suffix on them and provide the role's ARN in the RoleName variable. In addition to passing the IAM Role in the task, it is possible to set an IAM Role in the Service Instance Configuration by using the aws_iam_role property in the authentication section and providing the role's ARN.

This can also be referred to as role signing scenarios - need to have either a Task Role (roleName) and/or Adapter Role (aws_iam_role) to use ay of these scenarios.

Scenarios:

• IAM to internal AWS Server - either Task Role (roleName) or Adapter Role (aws_iam_role)
• Adapter Role (aws_iam_role) assumes Task Role (STSParams or roleName)
  • IAM to internal AWS Server for Adapter Role
  • AWS STS for assuming Task Role(s) using Adapter Role
• Pod Role assumes Adapter Role (aws_iam_role) assumes Task Role (STSParams, RoleName)
  • AWS STS for assuming Adapter Role using AWS Environment (Pod Role)
  • AWS STS for assuming Task Role(s) using Adapter Role

"authentication": {
  "auth_method": "aws_authentication",
  "aws_iam_role": "role_arn"
}

AMAZON STEPS FOR IAM ROLE

Increase number of hops if running IAP inside of docker on S3 instance

aws sso login --profile aws-bota-1
<export aws keys for CLI access>

aws ec2 modify-instance-metadata-options  --instance-id i-0e150236026b7c45d  --http-put-response-hop-limit 3 --http-endpoint enabled --region us-east-1

Create a new role and attach to it policies:

• go to your S3 instance, select it
• Actions->Security->Modify IAM Role
• Click 'Create New IAM Role'
• Create a role:

  Trusted entity type: AWS service
  Use Case: S3

Add desired policies to the role.

Save the role

Go back to your S3 instance and Actions->Security->Modify IAM Role, associate newly created role with your S3 instance

Troubleshooting

• Make sure you copied over the correct access key, secret key and session token.
• Turn on debug level logs for the adapter in IAP Admin Essentials.
• Turn on auth_logging for the adapter in IAP Admin Essentials (adapter properties).
• Investigate the logs - in particular:
  • The FULL REQUEST log to make sure the proper headers are being sent with the request.
  • The FULL BODY log to make sure the payload is accurate.
  • The CALL RETURN log to see what the other system is telling us.
• Credentials should be masked by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
• Remember when you are done to turn auth_logging off as you do not want to log credentials.
• For IAM, you can run this on the IAP server to verify you are getting to the "internal" AWS Server

  TOKEN=`curl -v -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -v -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/<rolename>

  Sample Properties

Sample Properties can be used to help you configure the adapter in the Itential Automation Platform. You will need to update connectivity information such as the host, port, protocol and credentials.

  "properties": {
    "host": "s3.amazonaws.com",
    "region": "us-east-1",
    "port": 443,
    "choosepath": "",
    "base_path": "/",
    "version": "",
    "cache_location": "none",
    "encode_pathvars": true,
    "encode_queryvars": true,
    "save_metric": false,
    "stub": true,
    "protocol": "https",
    "service": "s3",
    "authentication": {
      "auth_method": "aws_authentication",
      "username": "username",
      "password": "password",
      "token": "token",
      "token_timeout": 180000,
      "token_cache": "local",
      "invalid_token_error": 401,
      "auth_field": "header.headers.Cookie",
      "auth_field_format": "Token {token}",
      "auth_logging": false,
      "client_id": "",
      "client_secret": "",
      "grant_type": "",
      "sensitive": [],
      "sso": {
        "protocol": "",
        "host": "",
        "port": 0
      },
      "multiStepAuthCalls": [
        {
          "name": "",
          "requestFields": {},
          "responseFields": {},
          "successfullResponseCode": 200
        }
      ],
      "aws_access_key": "aws_access_key",
      "aws_secret_key": "aws_secret_key",
      "aws_session_token": "aws_session_token",
      "aws_iam_role": "",
      "aws_sts": {
        "region": "us-east-1",
        "sslEnable": true,
        "endpoint": "",
        "proxy": "",
        "proxyagent": "",
        "externalId": ""
      }
    },
    "healthcheck": {
      "type": "none",
      "frequency": 60000,
      "query_object": {},
      "addlHeaders": {}
    },
    "throttle": {
      "throttle_enabled": false,
      "number_pronghorns": 1,
      "sync_async": "sync",
      "max_in_queue": 1000,
      "concurrent_max": 1,
      "expire_timeout": 0,
      "avg_runtime": 200,
      "priorities": [
        {
          "value": 0,
          "percent": 100
        }
      ]
    },
    "request": {
      "number_redirects": 0,
      "number_retries": 3,
      "limit_retry_error": 0,
      "failover_codes": [],
      "attempt_timeout": 5000,
      "global_request": {
        "payload": {},
        "uriOptions": {},
        "addlHeaders": {},
        "authData": {}
      },
      "healthcheck_on_timeout": true,
      "return_raw": false,
      "archiving": false,
      "return_request": false
    },
    "proxy": {
      "enabled": false,
      "host": "",
      "port": 1,
      "protocol": "http",
      "username": "",
      "password": ""
    },
    "ssl": {
      "ecdhCurve": "",
      "enabled": false,
      "accept_invalid_cert": true,
      "ca_file": "",
      "key_file": "",
      "cert_file": "",
      "secure_protocol": "",
      "ciphers": ""
    },
    "mongo": {
      "host": "",
      "port": 0,
      "database": "",
      "username": "",
      "password": "",
      "replSet": "",
      "db_ssl": {
        "enabled": false,
        "accept_invalid_cert": false,
        "ca_file": "",
        "key_file": "",
        "cert_file": ""
      }
    },
    "devicebroker": {
      "enabled": false,
      "getDevice": [
        {
          "path": "/not/mapped",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "insample": "{port}"
          },
          "responseDatakey": "",
          "responseFields": {
            "name": "{this}{||}{that}",
            "ostype": "{osfield}",
            "ostypePrefix": "meraki-",
            "port": "{port}",
            "ipaddress": "{ip_addr}",
            "serial": "{serial}"
          }
        }
      ],
      "getDevicesFiltered": [
        {
          "path": "/not/mapped",
          "method": "GET",
          "pagination": {
            "offsetVar": "",
            "limitVar": "",
            "incrementBy": "limit",
            "requestLocation": "query"
          },
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {},
          "responseDatakey": "",
          "responseFields": {
            "name": "{this}{||}{that}",
            "ostype": "{osfield}",
            "ostypePrefix": "meraki-",
            "port": "{port}",
            "ipaddress": "{ip_addr}",
            "serial": "{serial}",
            "id": "{myid}"
          }
        }
      ],
      "isAlive": [
        {
          "path": "/not/mapped/{devID}",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "devID": "{id}"
          },
          "responseDatakey": "",
          "responseFields": {
            "status": "return2xx",
            "statusValue": "AD.200"
          }
        }
      ],
      "getConfig": [
        {
          "path": "/not/mapped/{devID}",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "devID": "{id}"
          },
          "responseDatakey": "",
          "responseFields": {}
        }
      ],
      "getCount": [
        {
          "path": "/not/mapped",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {},
          "responseDatakey": "",
          "responseFields": {}
        }
      ]
    },
    "cache": {
      "enabled": false,
      "entities": [
        {
          "entityType": "device",
          "frequency": 3600,
          "flushOnFail": false,
          "limit": 10000,
          "retryAttempts": 5,
          "sort": true,
          "populate": [
            {
              "path": "/not/mapped",
              "method": "GET",
              "pagination": {
                "offsetVar": "",
                "limitVar": "",
                "incrementBy": "limit",
                "requestLocation": "query"
              },
              "query": {},
              "body": {},
              "headers": {},
              "handleFailure": "ignore",
              "requestFields": {},
              "responseDatakey": "",
              "responseFields": {
                "name": "{this}{||}{that}",
                "ostype": "{osfield}",
                "ostypePrefix": "meraki-",
                "port": "{port}",
                "ipaddress": "{ip_addr}",
                "serial": "{serial}",
                "id": "{myid}"
              }
            }
          ],
          "cachedTasks": [
            {
              "name": "",
              "filterField": "",
              "filterLoc": ""
            }
          ]
        }
      ]
    }
  }

Swagger

Generic Adapter Information