# OpenTelemetry for Node.js on AWS Lambda

The Baselime Node.js OpenTelemetry tracer for AWS Lambda instruments your Node.js AWS Lambda functions with OpenTelemetry and automatically sends OpenTelemetry traces to Baselime. This is the most powerful and flexible way to instrument your Node.js AWS Lambda functions.


# Automatic Instrumentation

To automatically instrument your AWS Lambda functions with the Baselime Node.js OpenTelemetry tracer for AWS Lambda, set the following tag to your AWS Lambda functions: baselime:tracing=true.

To add the Baselime tag to all your AWS Lambda functions in a service or stack add this line to your AWS CDK code.

 Tags.of(app).add("baselime:tracing", `true`);

To add the Baselime tag to all your AWS Lambda functions in a service or stack add this line to your sst.config.ts file.

 Tags.of(app).add("baselime:tracing", `true`);

To add the Baselime tag to all your AWS Lambda functions in a add this snippet to your serverless.yml file.

provider:
  name: aws
  tags:
    "baselime:tracing": "true"

To add the Baselime tag to all your AWS Lambda functions in a add this snippet to your AWS SAM configuration file.

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: "Gets data from the xxxxx API."
Globals:
  Function:
    Tags:
       "baselime:tracing": "true"

That's all you are all set with OpenTelemetry on Node.js AWS Lambda functions.

# How it works

This section outlines how the automatic instrumentation works.

The automatic instrumentation makes changes to your AWS Lambda functions once they are deployed:

  • Add the Baselime Node.js OpenTelemetry AWS Lambda Layer to your AWS Lambda function: arn:aws:lambda:${region}:097948374213:layer:baselime-node:${version} - This layer is a slimmed down version of the OpenTelemetry JavaScript Client that will have minimal impact on the cold starts of your AWS Lambda functions
  • Add the Baselime Extension added to your AWS Lambda function: arn:aws:lambda:${region}:097948374213:layer:baselime-extension-${'x86_64' || 'arm64'}:${version} - This extension enables the Baselime Layers to send the trace data to the Baselime backend after the invocation is complete, as such, distributed tracing will not have any negative impact on the latency of your AWS Lambda functions
  • Set the BASELIME_KEY environment variable with the value of your environments Baselime API Key

These changes are kept in sync with your AWS Lambda function as you iterate on your architecture via events from Amazon CloudTrail.

OpenTelemetry Automatic Instrumentation FLow
OpenTelemetry Automatic Instrumentation FLow


# Adding custom OpenTelemetry events

The Baselime Node.js OpenTelemetry tracer for AWS Lambda provides and extension that enables you add context rich events to your traces using an API that feels like a logger. These events can be useful to show more detailed context on errors, add steps that you want recorded for a business process or adding extra debugging information.

const { logger } = require("@baselime/lambda-node-opentelemetry");

logger.info("This is an informational message", {
  operation: "copy-paste-replace",
  count: 9000,
});

The extension provides an object that includes four logging functions - info, warn, debug, and error - enabling you to log messages with varying levels of severity. You can control the visibility of the events by setting the LOG_LEVEL environment variable, .

const { logger } = require("@baselime/lambda-node-opentelemetry");

logger.info("This is an informational message", { payload: { foo: "bar" } });
logger.warn("This is a warning message", { payload: { foo: "bar" } });
logger.debug("This is a debug message", { payload: { foo: "bar" } });
logger.error("This is an error message", { payload: { foo: "bar" } });

# Adding custom OpenTelemetry spans

To add custom spans to your OpenTelemetry traces, it is necessary to install the @opentelemetry/api package. It is left out of the Baselime Node.js OpenTelemetry tracer for AWS Lambda to limit the impact on cold-starts, such that your can add it only to the AWS Lambda functions that require it.

import { trace } from "@opentelemetry/api";
const tracer = trace.getTracer('your-custom-traces');

export async function handler(event) {
  const activeSpan = trace.getActiveSpan();
  
  const { userId } = JSON.parse(event.body);
  span.setAttribute('user', userId)
  
  // do something meaningful
  
  const result = await tracer.startActiveSpan(`business-logic`, async (span) => {
    span.setAttributes(args)
    // your business logic
    const result = await yourBusinessLogic(args)
    span.setAttributes(result)
    return result
  });
}

# Manual Instrumentation

To manually instrument your Node.js AWS Lambda functions, you can use the Baselime Node.js OpenTelemetry tracer for AWS Lambda independently from connecting your AWS Account to Baselime.

# Step 1: Install the SDK

Install the Baselime Node.js OpenTelemetry tracer for AWS Lambda.

npm install @baselime/lambda-node-opentelemetry

# Step 2: Manually wrap your function handlers with the SDK

Wrap the handlers of your AWS Lambda functions with the baselime.wrap(handler) method.

import baselime from '@baselime/lambda-node-opentelemetry'

async function main(event, context) {
   // your handler logic
}

export const handler = baselime.wrap(main);

# Step 3: Set the Baselime environment variables

Set the environment variables of your AWS Lambda functions to include the Baselime API Key and set the NODE_OPTIONS enviroment variable to preload the OpenTelemetry SDK into your AWS Lambda bundle.

Key Value Description
BASELIME_KEY your-api-key Get this key from the Baselime console or the Baselime CLI running baselime iam
NODE_OPTIONS --require @baselime/lambda-node-opentelemetry Preloads the OpenTelemetry SDK at startup

# Step 4: Bundle the OpenTelemetry SDK with your code

Ensure that the OpenTelemetry SDK is included in the .zip file that is uploaded to AWS Lambda during your deployment. The step depends on your deployment framework.

Set the default function props of your service to include the wrapper in the bundle and add the environment variables

app.setDefaultFunctionProps({
  runtime: "nodejs18.x",
  environment: {
    NODE_OPTIONS: '--require @baselime/lambda-node-opentelemetry/lambda-wrapper',
    BASELIME_KEY: process.env.BASELIME_KEY
  },
  nodejs: {
    install: ["@baselime/lambda-node-opentelemetry"],
  },
});

By default the Serverless Framework includes the entire node_module folder in the .zip bundle of your AWS Lambda functions. If you are using the serverless-esbuild plugin or any other plugin to prevent this, it is necessary to edit the configuration of your project.

Add the following line to the package.patterns block of your serverless.yml file.

package:
  patterns:
    - 'node_modules/@baselime/lambda-node-opentelemetry'

Add the following environment variables

    BASELIME_KEY: ${env:BASELIME_KEY}
    NODE_OPTIONS: '--require @baselime/lambda-node-opentelemetry/lambda-wrapper'

Copy the lambda-wrapper.js file from the node_modules folder in the shared folder of your Architect project, it will be automatically included in all of your AWS Lambda .zip bundles.

cp node_modules/@baselime/lambda-node-opentelemetry/lambda-wrapper.js src/shared/

Add the environment variables to your architect project

arc env -e production --add BASELIME_KEY <your-api-key>
arc env -e production --add -- NODE_OPTIONS '--require @architect/shared/lambda-wrapper'

Note the '--' in the NODE_OPTIONS command. This is required to escape options parsing.

This method will however send the traces to the Baselime backend during the invocation of your AWS Lambda functions, and will result in a degradation in the latency performace of your functions.

In production we recommend additionally adding the Baselime AWS Lambda extension, as it will enable the OpenTelemetry tracer to send traces to the Baselime backend after the excecution of your AWS Lambda functions.

`arn:aws:lambda:${region}:097948374213:layer:baselime-extension-${'x86_64' || 'arm64'}:1`

# Sending data to another OpenTelemetry backend

OpenTelemetry is an open standard, and you can use the Baselime Node.js OpenTelemetry tracer for AWS Lambda to send telemetry data to another backend of your choice.

Add the environment variable COLLECTOR_URL to send the data somewhere else than the Baselime backend.


# Limitations

The AWS JS SDK v2 can result in errors when interacting with OpenTelemetry during automatic request retries. This is the result of trace headers changing between retries and failing the signing verification processes. We've submitted a Pull Request to the AWS JS SDK and will be updating accordingly.

To prevent this issue from arising, add the code snippet below to your code.

const SignersV4 = require('aws-sdk/lib/signers/v4')

SignersV4.prototype.unsignableHeaders.push('traceparent');