# OpenTelemetry for Node.js on AWS Lambda

The Baselime Node.js OpenTelemetry tracer for AWS Lambda instruments your Node.js AWS Lambda functions with OpenTelemetry and automatically sends OpenTelemetry traces to Baselime. This is the most powerful and flexible way to instrument your Node.js AWS Lambda functions.

# Automatic Instrumentation

To automatically instrument your AWS Lambda functions with the Baselime Node.js OpenTelemetry tracer for AWS Lambda, set the following tag to your AWS Lambda functions: baselime:tracing=true.

For detailed instructions on how to add the tag to for your framework go to the OpenTelemetry for AWS Lambda Guide

# Configuration

Our OpenTelemetry SDK can be configured with the following environment variables. For each value other than log level set it to 'true' to turn the feature on.

Environment Variable	Description	Default Value
BASELIME_REQUEST_CAPTURE	Controls capturing of incoming requests	Not set
AWS_SDK_INTERNALS	Controls whether AWS SDK internals are logged	Not set
BASELIME_CAPTURE_EVENT	Controls capturing of specific events	Not set
BASELIME_CAPTURE_RESPONSE	Controls capturing of outgoing responses	Not set
OTEL_LOG_LEVEL	Sets the logging level for OpenTelemetry instrumentation. Set to "debug" to see granular diagnostics	Not set

# Adding custom OpenTelemetry spans

To add custom spans to your OpenTelemetry traces, install the @opentelemetry/api package.

npm i --save @opentelemetry/api

And manually add spans to your traces.

import { trace } from "@opentelemetry/api";
const tracer = trace.getTracer('your-custom-traces');

export async function handler(event) {
  const activeSpan = trace.getActiveSpan();
  
  const { userId } = JSON.parse(event.body);
  span.setAttribute('user', userId)
  
  // do something meaningful
  
  const result = await tracer.startActiveSpan(`business-logic`, async (span) => {
    span.setAttributes(args)
    // your business logic
    const result = await yourBusinessLogic(args)
    span.setAttributes(result)
    return result
  });
}

# Tracing AWS SDK v3

The Node.js AWS SDK v3 is not traced by default using OpenTelemetry on AWS Lambda when bundled as part of your function code. We currently support instrumentation for CommonJS codebases. We currently support instrumentation for CommonJS codebases. Use the following steps to enable tracing of the Node.js AWS SDK v3.

Mark @smithy/middleware-stack and @aws-sdk/middleware-stack as external.
Ensure your functions .zip file includes these dependencies

These packages are both extremely small and removing these from your bundle can also decrease your coldstarts

Follow the example below for popular serverless frameworks

Add the following config to your lambda.NodejsFunction settings

new lambda.NodejsFunction(this, 'my-handler', {
  bundling: {
    nodeModules: ['@smithy/middleware-stack', '@aws-sdk/middleware-stack'],
    format: lambda.OutputFormat.CJS
  },
});

Add the config globally to your sst.config.ts file.

app.setDefaultFunctionProps({
  nodejs: {
    format: 'cjs',
    install: ["@smithy/middleware-stack", "@aws-sdk/middleware-stack"]
  },
});

If using the serverless-esbuild plugin set the following options in your serverless.yml

plugins:
  - serverless-esbuild

custom:
  esbuild:
    external:
      - "@smithy/middleware-stack"
      - "@aws-sdk/middleware-stack"

# Step functions

Traces can be propogated accross Lambda invocations in step functions by adding the environment variable BASELIME_TRACE_STEP_FUNCTIONS.

# Propogation

Lambda functions will automatically propogate the trace state between steps by attaching the _baselime tracing metadata object to your functions response. This will be automatically detected if the lambda payloads are not modified.

If you are using custom parameters update the functions input to include the following.

Using the AWS CDK manually propagate the _baselime tracing metadata using the following code snippet

 const taskTwoA = new LambdaInvoke(stack, "TaskTwoA", {
    lambdaFunction: new Function(stack, "task-two-a", {
      handler: "packages/functions/src/task-two.handler",
    }),
    payload: TaskInput.fromObject({
      code: TaskInput.fromJsonPathAt("$.Payload.statusCode").value,
      _baselime: TaskInput.fromJsonPathAt("$.Payload._baselime").value
    })
  })

Using the Amazon States Language manually propagate the _baselime tracing metadata using the following code snippet

{
    "StartAt": "TaskTwoA",
    "States": {
      "TaskTwoA": {
        "End": true,
        "Type": "Task",
        "Resource": "arn:aws:states:::lambda:invoke",
        "Parameters": {
          "FunctionName": "arn:aws:lambda:us-east-1:123456789:function:prod-state-machine-lambda-time",
          "Payload": {
            "code.$": "$.Payload.statusCode",
            "_baselime.$": "$.Payload._baselime"
          }
        }
      }
    }
}

If you have tasks which cannot be instrumented like SNS: Publish then propogate the traceparent with the result path result path so the Baselime state is appended to the output of the task.

Using the AWS CDK manually propagate the _baselime tracing metadata using the following code snippet

const snsPublish = new SnsPublish(stack, "SnsPublish", {
    topic: SNS.fromTopicArn(stack, "SnsTopic", "arn:aws:sns:us-east-1:123456789012:MyTopic"),
    message: TaskInput.fromObject({
      ...
    }),
    resultPath: "$.Payload._baselime",
})

Using the Amazon States Language manually propagate the _baselime tracing metadata using the following code snippet

{
  "SNS Publish": {
  "Type": "Task",
  "Resource": "arn:aws:states:::sns:publish",
  "Parameters": {
    "Message.$": "$",
    "TopicArn": "arn:aws:sns::1249256823:topic:my-topic"
  },
  "Next": "TaskTwoA",
  "ResultPath": "$.Payload._baselime"
}

# Sending data to another OpenTelemetry backend

OpenTelemetry is an open standard, and you can use the Baselime Node.js OpenTelemetry tracer for AWS Lambda to send telemetry data to another backend of your choice.

Set environment variable COLLECTOR_URL to your observability backend.

# Limitations

The AWS JS SDK v2 can result in errors when interacting with OpenTelemetry during automatic request retries. This is the result of trace headers changing between retries and failing the signing verification processes. We've submitted a Pull Request to the AWS JS SDK and will be updating accordingly.

To prevent this issue from arising, add the code snippet below to your code.

const SignersV4 = require('aws-sdk/lib/signers/v4')

SignersV4.prototype.unsignableHeaders.push('traceparent');