Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(aws-serverless): Extract sentry trace data from handler context over event #13266

Merged
merged 4 commits into from
Aug 8, 2024

Conversation

andreiborza
Copy link
Member

@andreiborza andreiborza commented Aug 7, 2024

Currently, the AWS otel integration (and our wrapHandler fallback) try to extract sentry trace data from the event object passed to a Lambda call. The aws-sdk integration, however, places tracing data onto context.clientContext.Custom.

This PR adds a custom eventContextExtractor that attempts extracting sentry trace data from the context, with a fallback to event to enable distributed tracing among Lambda invocations.

Traces are now connected. Here an example:

Lambda-A calling Lambda-B:

import { LambdaClient, InvokeCommand } from "@aws-sdk/client-lambda";
import * as Sentry from "@sentry/aws-serverless";

export const handler = Sentry.wrapHandler(async (event, context) => {
  const client = new LambdaClient();
  const command = new InvokeCommand({
    FunctionName: `Lambda-B`,
    InvocationType: "RequestResponse",
    Payload: new Uint16Array(),
  })
  return client.send(command);

});

Lambda-B:

import * as Sentry from "@sentry/aws-serverless";
Sentry.addIntegration(Sentry.postgresIntegration())

export const handler = Sentry.wrapHandler(async (event) => {
  const queryString = "select count(*) from myTable;";
  return await Sentry.startSpan({
    name: queryString,
    op: "db.sql.execute"
  }, async (span) => {
    console.log('executing query', queryString);
  })
})

CleanShot 2024-08-07 at 16 34 51@2x

Closes: #13146

Copy link
Contributor

github-actions bot commented Aug 7, 2024

size-limit report 📦

Path Size
@sentry/browser 22.47 KB (0%)
@sentry/browser (incl. Tracing) 34.26 KB (0%)
@sentry/browser (incl. Tracing, Replay) 70.31 KB (0%)
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 63.64 KB (0%)
@sentry/browser (incl. Tracing, Replay with Canvas) 74.71 KB (0%)
@sentry/browser (incl. Tracing, Replay, Feedback) 87.32 KB (0%)
@sentry/browser (incl. Tracing, Replay, Feedback, metrics) 89.16 KB (0%)
@sentry/browser (incl. metrics) 26.77 KB (0%)
@sentry/browser (incl. Feedback) 39.42 KB (0%)
@sentry/browser (incl. sendFeedback) 27.09 KB (0%)
@sentry/browser (incl. FeedbackAsync) 31.75 KB (0%)
@sentry/react 25.24 KB (0%)
@sentry/react (incl. Tracing) 37.25 KB (0%)
@sentry/vue 26.62 KB (0%)
@sentry/vue (incl. Tracing) 36.1 KB (0%)
@sentry/svelte 22.61 KB (0%)
CDN Bundle 23.69 KB (0%)
CDN Bundle (incl. Tracing) 35.93 KB (0%)
CDN Bundle (incl. Tracing, Replay) 70.36 KB (0%)
CDN Bundle (incl. Tracing, Replay, Feedback) 75.61 KB (0%)
CDN Bundle - uncompressed 69.52 KB (0%)
CDN Bundle (incl. Tracing) - uncompressed 106.43 KB (0%)
CDN Bundle (incl. Tracing, Replay) - uncompressed 218.27 KB (0%)
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 231.16 KB (0%)
@sentry/nextjs (client) 37.1 KB (0%)
@sentry/sveltekit (client) 34.81 KB (0%)
@sentry/node 114.85 KB (+0.01% 🔺)
@sentry/node - without tracing 89.33 KB (0%)
@sentry/aws-serverless 98.75 KB (+0.26% 🔺)

@@ -9,6 +9,10 @@ export default [
entrypoints: ['src/index.ts', 'src/awslambda-auto.ts'],
// packages with bundles have a different build directory structure
hasBundles: true,
packageSpecificConfig: {
// Used for our custom eventContextExtractor
external: ['@opentelemetry/api'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just out of curiosity, why is this needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Package build structure isn't right without this. It pulls this in under its own node_modules folder.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment on lines 10 to 11
* Sentry's context. This should usually be `true` when
* using Sentry to instrument Lambdas.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Sentry's context. This should usually be `true` when
* using Sentry to instrument Lambdas.
* Sentry's context. Defaults to `true`, in order for Sentry trace propagation to take precedence,
* but can be disabled if you want to AWS propagation to take precedence.

or something like this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated :)

@@ -25,7 +25,7 @@ import { SEMANTIC_ATTRIBUTE_SENTRY_ORIGIN, SEMANTIC_ATTRIBUTE_SENTRY_SOURCE } fr
import { DEBUG_BUILD } from './debug-build';
import { awsIntegration } from './integration/aws';
import { awsLambdaIntegration } from './integration/awslambda';
import { markEventUnhandled } from './utils';
import { getTraceData, markEventUnhandled } from './utils';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: Can we change the name here, as it conflicts with getTraceData we have in core, making this possibly a bit confusing 😅 E.g. getAwsTraceData or something like this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, good point.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

const customContext: Record<string, unknown> = context.clientContext.Custom;
const sentryTrace = isString(customContext['sentry-trace']) ? customContext['sentry-trace'] : undefined;

if (sentryTrace) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: It will probably not matter in 99% of cases, but let's just split this into two if-blocks. So basically:

const sentryTrace = ...;
const baggage = ...;

if(sentryTrace) {
  traceData['sentry-trace'] = sentryTrace;
}
if(baggage) {
  traceData.baggage = baggage;
}

I think this is slightly more robust :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking of that, but wouldn't we potentially end up with a mix of wrong pairs, e.g. both event and context have baggage but only event has sentry-trace. So we end up with sentry-trace from event and baggage from context.

I don't know if that's a realistic usecase tho. In core, we also throw away baggage if sentryTrace is not valid.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, then let's keep it this way, all good! 👍


/**
* Extracts sentry trace data from the handler `context` if available and falls
* back to the `event`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can also add a short note here when this would be on context and when on event?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an explanation, but tbh this is pretty wide open. I think different AWS services make use of different event/context usage. Hope this adds a bit more info?

Copy link
Member

@mydea mydea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some notes, more about details - overall great work! Big improvement :)

@andreiborza andreiborza requested a review from mydea August 8, 2024 07:53
@andreiborza andreiborza merged commit b17ac59 into develop Aug 8, 2024
101 checks passed
@andreiborza andreiborza deleted the ab/aws-lambda-client-context branch August 8, 2024 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Spans in continueTrace callback are not being recorded
2 participants