# AWS Connector on Baselime

The AWS Connector allows you to send data from your AWS resources to Baselime. This includes logs, traces, and metrics. By connecting your AWS account to Baselime, you can get a unified view of your architecture, query your data, and set up alerts.

# Setting up the AWS Connector

The connector is an automated flow based on a CloudFormation template.

It can be done using the Baselime CLI or through the web console.

# Using the CLI

To connect a cloud account to Baselime using the CLI, run the following command in your terminal

 terminal
baselime environments connect --provider aws --account <account_numner> --region <region> --alias <alias>

Once you've followed the interactive steps, the CLI will generate a CloudFormation template for you to deploy on your AWS account. FOllow the link in your terminal to deploy the temple on your AWS account.

Once deployed, login in your newly connected environment from the CLI.

 terminal
baselime login

The interactive prompt should list your newly connected environment.

Within minutes you should get telemetry data flowing through with the command

 terminal
baselime tail --follow

# Using the Web Console

Navigate to the Baselime web console and login.

Follow the steps on the homescreen to connect a new AWS Account. Baselime will generate a CloudFormation template for you to deploy on your AWS account.

Once the template is deployed on AWS, return to the Baselime web console and refresh the page. You should see the newly connected AWS environment in the list of connected environment.

Within minutes telemetry data from your AWS environment should start displaying in the events streams in the Baselime web console.

# Troubleshooting

If you encounter any issues or error when connecting your AWS environment, please don't hesitate to contact us, or join our Slack community where we are always available to support.

# CloudFormation Template

The CloudFormation template is open-source and available here.

Baselime AWS Integration Template
Description: This template creates the resources necessary for Baselime to observe your AWS Account

Parameters:
  ExternalParameter:
    Type: String
    Default: <BASELIME_WORKSPACE_ID>
    Description: External Parameter for securing the role
  Alias:
    Type: String
    Default: <BASELIME_ENVIRONMENT_ALIAS>
    Description: Alias for this environment

Resources:
  ###
  # Role that Baselime can assume to perform actions on your AWS Account
  # Those actions are limited to read-only operations and operations on resources prefixed with 'baselime'
  ###
  BaselimeEnvironmentRole:
    Type: AWS::IAM::Role
    DeletionPolicy: Retain
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              AWS:
                - <BASELIME_CUSTOMER_ACCOUNT>
            Sid: ""
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              AWS:
                - <BASELIME_ACCOUNT>
            Sid: ""
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
            Sid: ""
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - events.amazonaws.com
            Sid: ""
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - appsync.amazonaws.com
            Sid: ""
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/ReadOnlyAccess
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/service-role/AWSAppSyncPushToCloudWatchLogs
        - arn:aws:iam::aws:policy/AWSAppSyncAdministrator
      Path: "/"
  BaselimeEnvironmentPolicies:
    Type: AWS::IAM::Policy
    DeletionPolicy: Retain
    Properties:
      PolicyName: BaselimeEnvironment
      PolicyDocument:
        Statement:
          - Effect: Allow
            # Those operations are limited to Lambda functions prefixed with 'baselime'
            Action:
              - lambda:CreateFunction
              - lambda:AddPermission
              - lambda:DeleteFunction
              - lambda:UpdateFunctionConfiguration
              - lambda:UpdateFunctionCode
            Resource:
              - arn:aws:lambda:*:*:function:baselime-*
          - Effect: Allow
            Action:
              - lambda:UpdateFunctionConfiguration
            Resource:
              - arn:aws:lambda:*:*:function:*
          - Effect: Allow
            Action:
              - ecs:UpdateService
            Resource:
              - arn:aws:ecs:*:*:service/*/*
          - Effect: Allow
            Action:
              - ecs:RegisterTaskDefinition
            Resource:
              - "*"
          - Effect: Allow
            # Those operations are limited to CloudFormation stacks prefixed with 'baselime'
            Action:
              - cloudformation:*
            Resource:
              - arn:aws:cloudformation:*:*:stack/Baselime*
              - !Ref AWS::StackId
          - Effect: Allow
            # Those operations are limited to the CloudTrail created as part of this CF stack
            Action:
              - cloudtrail:*
            Resource:
              - Fn::GetAtt:
                - BaselimeCloudTrail
                - Arn
          - Effect: Allow
            # Those operations are limited to the S3 bucket created as part of this CF stack
            Action:
              - s3:*
            Resource:
              - !Sub |-
                arn:aws:s3:::${BaselimeS3Bucket}
              - !Sub |-
                arn:aws:s3:::${BaselimeS3Bucket}/*
          - Effect: Allow
            # Those operations are limited to the SNS topic created as part of this CF stack
            Action:
              - sns:*
            Resource:
              - !Ref BaselimeSNSTopic
          - Effect: Allow
            # This operation is limited to a Lambda function on Baselime's Account
            Action:
              - lambda:InvokeFunction
            Resource:
              - <BASELIME_SERVICE_TOKEN>
          - Effect: Allow
            Action:
              - ce:GetCostAndUsageWithResources
              - ce:GetCostAndUsage
            Resource:
              - "*"
          - Effect: Allow
            # This operation is to enable Baselime to use this role
            Action:
              - iam:PassRole
            Resource:
              - !GetAtt
                - BaselimeEnvironmentRole
                - Arn
          - Effect: Allow
            Action:
              - events:PutEvents
            Resource:
              - <BASELIME_COMMON_EVENTBUS_ARN>
          - Effect: Allow
            Action:
              - logs:PutSubscriptionFilter
              - logs:DeleteSubscriptionFilter
              - logs:DescribeSubscriptionFilters
              - logs:CreateLogGroup
              - logs:PutRetentionPolicy
              - cloudwatch:PutMetricAlarm
              - cloudwatch:DeleteAlarms
              - cloudwatch:PutDashboard
              - cloudwatch:DeleteDashboards
              - cloudwatch:PutMetricData
              - cloudwatch:PutMetricStream
              - cloudwatch:DeleteMetricStream
              - cloudwatch:StartMetricStreams
              - cloudwatch:StopMetricStreams
              - events:PutRule
              - events:PutTargets
              - events:DeleteRule
              - events:RemoveTargets
            Resource:
              - "*"
          - Effect: Allow
            # These operations are limited to the Kinesis Firehose created as part of this CF Stack
            Action:
              - firehose:DescribeDeliveryStream
              - firehose:DeleteDeliveryStream
              - firehose:UpdateDestination
            Resource:
              - Fn::GetAtt:
                - BaselimeCloudWatchMetricsFirehose
                - Arn
              - Fn::GetAtt:
                - BaselimeRawDataFirehose
                - Arn
          - Effect: Allow
            Action:
              - firehose:PutRecord
            Resource:
              - Fn::GetAtt:
                  - BaselimeRawDataFirehose
                  - Arn
          # These operations are limited to the other IAM Roles created as part of this CF Stack
          - Effect: Allow
            Action:
              - iam:DeleteRolePolicy
              - iam:DeleteRole
            Resource:
              - Fn::GetAtt:
                - BaselimeMetricsStreamFirehoseRole
                - Arn
              - Fn::GetAtt:
                - BaselimeRawDataFirehoseRole
                - Arn
              - Fn::GetAtt:
                - BaselimeCloudWatchMetricsStreamRole
                - Arn
      Roles:
       - !Ref BaselimeEnvironmentRole
  ###
  # S3 Bucket that Baselime uses to store CloudTrail logs from your account
  ###
  BaselimeS3Bucket:
    Type: 'AWS::S3::Bucket'
    DeletionPolicy: Retain
    Properties:
      LifecycleConfiguration:
        Rules:
          - Id: RawData
            Status: Enabled
            Prefix: RawData/
            ExpirationInDays: 182
  BaselimeS3BucketPolicy:
    Type: "AWS::S3::BucketPolicy"
    Properties:
      Bucket:
        Ref: BaselimeS3Bucket
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          -
            Sid: "AWSCloudTrailAclCheck"
            Effect: "Allow"
            Principal:
              Service: "cloudtrail.amazonaws.com"
            Action: "s3:GetBucketAcl"
            Resource:
              !Sub |-
                arn:aws:s3:::${BaselimeS3Bucket}
          -
            Sid: "AWSCloudTrailWrite"
            Effect: "Allow"
            Principal:
              Service: "cloudtrail.amazonaws.com"
            Action: "s3:PutObject"
            Resource:
              !Sub |-
                arn:aws:s3:::${BaselimeS3Bucket}/AWSLogs/${AWS::AccountId}/*
            Condition:
              StringEquals:
                s3:x-amz-acl: "bucket-owner-full-control"
  ###
  # SNS Topic that Baselime uses to notify when there's new CloudTrail data available on your account
  ###
  BaselimeSNSTopic:
    Type: AWS::SNS::Topic
  BaselimeSNSTopicPolicy:
    Type: "AWS::SNS::TopicPolicy"
    Properties:
      Topics:
        - !Ref BaselimeSNSTopic
      PolicyDocument:
        Version: "2008-10-17"
        Statement:
          -
            Sid: "AWSCloudTrailSNSPolicy"
            Effect: "Allow"
            Principal:
              Service: "cloudtrail.amazonaws.com"
            Resource: "*"
            Action: "SNS:Publish"
          -
            Sid: "BaselimeRoleSNSPolicy"
            Effect: "Allow"
            Principal:
              AWS: !GetAtt
              - BaselimeEnvironmentRole
              - Arn
            Resource: "*"
            Action: "SNS:Publish"
  ###
  # CloudTrail trail that Baselime uses to provide you with Observability on the actions across your AWS Account
  # You can disable this resource at any point from the Baselime console
  ###
  BaselimeCloudTrail:
    Type: AWS::CloudTrail::Trail
    DependsOn:
      - BaselimeSNSTopicPolicy
      - BaselimeS3BucketPolicy
    Properties:
        S3BucketName: !Ref BaselimeS3Bucket
        SnsTopicName: !GetAtt
                      - BaselimeSNSTopic
                      - TopicName
        IsLogging: true
        EnableLogFileValidation: true
        IncludeGlobalServiceEvents: true
        EventSelectors:
          - ReadWriteType: WriteOnly
  ###
  # Custom CloudFormation resource to notify Baselime when this CloudFormation stack is created, updated or deleted
  ###
  BaselimeReporter:
    Type: AWS::CloudFormation::CustomResource
    DependsOn:
      - BaselimeCloudTrail
      - BaselimeEnvironmentRole
      - BaselimeEnvironmentPolicies
      - BaselimeCloudWatchMetricsFirehose
      - BaselimeCloudWatchMetricsStream
    Properties:
      ServiceToken: <BASELIME_SERVICE_TOKEN>
      RoleArn: !GetAtt
              - BaselimeEnvironmentRole
              - Arn
      Region: !Ref AWS::Region
      StackName: !Ref AWS::StackName
      BucketName: !Ref BaselimeS3Bucket
      TopicArn: !Ref BaselimeSNSTopic
      CloudTrailName: !Ref BaselimeCloudTrail
      ExternalParameter: !Ref ExternalParameter
      Alias: !Ref Alias
      CloudWatchMetricsFirehoseName: !Ref BaselimeCloudWatchMetricsFirehose
      CloudWatchMetricsStreamName: !Ref BaselimeCloudWatchMetricsStream
      RawDataFirehoseName: !Ref BaselimeRawDataFirehose

  ###
  # Resources necessary for Baselime to stream CloudWatch Metrics
  # These resources comprise
  # - 2 IAM Roles
  # - A CloudWatch Metrics Stream
  # - A Kinesis Firehose Role
  # You can disable these resources at any point from the Baselime console
  ###
  BaselimeCloudWatchMetricsStreamRole:
    Type: AWS::IAM::Role
    DependsOn:
      - BaselimeCloudWatchMetricsFirehose
    Properties:
      Description: Role used by Kinesis Firehose to push data to Baselime
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service: streams.metrics.cloudwatch.amazonaws.com
      Policies:
        - PolicyName: BaselimeCloudWatchMetricsFirehosePutRecords
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - firehose:PutRecord
                  - firehose:PutRecordBatch
                Resource:
                  - Fn::GetAtt:
                      - BaselimeCloudWatchMetricsFirehose
                      - Arn
  BaselimeCloudWatchMetricsStream:
    Type: AWS::CloudWatch::MetricStream
    DependsOn:
      - BaselimeCloudWatchMetricsFirehose
      - BaselimeCloudWatchMetricsStreamRole
    Properties:
      FirehoseArn:
        Fn::GetAtt:
          - BaselimeCloudWatchMetricsFirehose
          - Arn
      OutputFormat: json
      RoleArn:
        Fn::GetAtt:
          - BaselimeCloudWatchMetricsStreamRole
          - Arn
  BaselimeRawDataFirehoseRole:
    Type: AWS::IAM::Role
    DependsOn:
      - BaselimeS3Bucket
    Properties:
      Description: Role used by Kinesis Firehose to push raw data to s3
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service: firehose.amazonaws.com
      Policies:
        - PolicyName: BaselimeRawDataFirehoseRoleServicePolicy
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - s3:AbortMultipartUpload
                  - s3:GetBucketLocation
                  - s3:GetObject
                  - s3:ListBucket
                  - s3:ListBucketMultipartUploads
                  - s3:PutObject
                Resource:
                  - !Sub |-
                    arn:aws:s3:::${BaselimeS3Bucket}
                  - !Sub |-
                    arn:aws:s3:::${BaselimeS3Bucket}/*

  BaselimeRawDataFirehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    DependsOn:
      - BaselimeRawDataFirehoseRole
      - BaselimeS3Bucket
    Properties:
      DeliveryStreamType: "DirectPut"
      ExtendedS3DestinationConfiguration:
        Prefix: "baselime-raw/data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"
        ErrorOutputPrefix: "baselime-raw/errors/type=!{firehose:error-output-type}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"
        BucketArn:
          Fn::GetAtt:
            - BaselimeS3Bucket
            - Arn
        RoleARN:
          Fn::GetAtt:
            - BaselimeRawDataFirehoseRole
            - Arn
  BaselimeCloudWatchMetricsFirehose:
    Type: AWS::KinesisFirehose::DeliveryStream
    DependsOn:
      - BaselimeMetricsStreamFirehoseRole
      - BaselimeS3Bucket
    Properties:
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          AccessKey: !Join
            - '-'
            - - '<BASELIME_CUSTOMER_ACCOUNT>'
              - !Ref AWS::Region
              - '<BASELIME_WORKSPACE_ID>'
              - '<BASELIME_ENVIRONMENT_ALIAS>'
          Name: BaselimeCloudwatchMetricsEndpoint
          Url: <BASELIME_CLOUDWATCH_METRICS_STREAM_ENDPOINT>
        RetryOptions:
          DurationInSeconds: 300
        S3Configuration:
          BucketARN:
            Fn::GetAtt:
              - BaselimeS3Bucket
              - Arn
          Prefix: "firehose-cloudwatch-metrics"
          RoleARN:
            Fn::GetAtt:
              - BaselimeMetricsStreamFirehoseRole
              - Arn
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 5
        RoleARN:
          Fn::GetAtt:
            - BaselimeMetricsStreamFirehoseRole
            - Arn
  BaselimeMetricsStreamFirehoseRole:
    Type: AWS::IAM::Role
    Properties:
      Description: Role used by Kinesis Firehose to push data to Baselime
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service: firehose.amazonaws.com
      Policies:
        - PolicyName: BaselimeMetricsStreamFirehoseServicePolicy
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - s3:AbortMultipartUpload
                  - s3:GetBucketLocation
                  - s3:GetObject
                  - s3:ListBucket
                  - s3:ListBucketMultipartUploads
                  - s3:PutObject
                Resource:
                  - !Sub |-
                    arn:aws:s3:::${BaselimeS3Bucket}
                  - !Sub |-
                    arn:aws:s3:::${BaselimeS3Bucket}/*


Outputs:
  RoleArn:
    Value: !GetAtt
          - BaselimeEnvironmentRole
          - Arn
    Description: The ARN for the role Baselime can use
  BucketName:
    Value: !Ref BaselimeS3Bucket
    Description: The name of the bucket Baselime can use
  SNSTopic:
    Value: !Ref BaselimeSNSTopic
    Description: The ARN of the SNS topic Baselime can use

# Your data

Once connected, Baselime will automatically ingest data from your AWS environment. This includes:

Lambda Logs
API Gateway Logs
Cloudtrail Logs
Cloudwatch Metrics
ECS Logs (through fluentd)
Open Telemetry Metrics
X-Ray Traces

Once ingested, the telemetry data is streamed through a Kinesis Firehose to an Amazon S3 bucket in your AWS account for cold storage. There you can access the raw data and use it for your own purposes.

The default retention period of the telemetry data in your bucket is set to 180 days by default.