feat: add request layer lifecycle TracingChannels

[logaretm]

This PR adds a single tracing channel that emits structured lifecycle events (start, end, asyncStart, asyncEnd, error) for every middleware, route handler, and error handler execution.

This builds on #96 but uses tracing channels instead which enables context propagation out of the box.

Context shape

The context object is pretty minimal, providing both req and res enables APMs to figure out how to react to that specific payload.

{ req, res, layer }

The layer is the Layer instance. Subscribers use layer.name, layer.handle.length (4 for error handlers), and req.route (set when inside a matched route) to classify and name spans however they see fit.

Usage Example:

This is an example I've set up locally using a Sentry setup with this PR patched in:

const Sentry = require('@sentry/node')
// otel-tracing-channel is a simple tracing channel wrapper that patches in the OTEL context propagation
// atm otel doesn't support tracing channels, so this workaround is required for OTEL-based APMs
// the user wouldn't need this and typically they can set their own thing with ALS.
const { tracingChannel } = require('otel-tracing-channel')

Sentry.init({
  // ...
})

// Subscribe to express:request TracingChannel and create Sentry spans
const channel = tracingChannel('pillarjs.router.request', (ctx) => {
  const { layer, req } = ctx
  const isErrorHandler = layer.handle.length === 4
  const isRouteHandler = Boolean(req.route)

  const spanName = isRouteHandler
    ? `${req.method} ${req.route.path}`
    : isErrorHandler
      ? `error_handler - ${layer.name}`
      : `middleware - ${layer.name}`

  return Sentry.startSpanManual(
    {
      name: spanName,
      op: isRouteHandler ? 'http.handler' : isErrorHandler ? 'middleware.error' : 'middleware',
      attributes: {
        'express.name': layer.name,
        'http.request.method': req.method,
        'url.path': req.url,
        'url.full': req.originalUrl,
        ...(req.route && { 'http.route': req.route.path }),
        ...(req.query && Object.keys(req.query).length > 0 && {
          'url.query': JSON.stringify(req.query)
        }),
        ...(req.params && Object.keys(req.params).length > 0 && {
          'express.params': JSON.stringify(req.params)
        })
      }
    },
    (span) => span
  )
})

channel.subscribe({
  asyncEnd (ctx) {
    if (ctx.req.route) {
      ctx.span?.setAttribute('http.response.status_code', ctx.res.statusCode)
    }
    ctx.span?.end()
  },
  error (ctx) {
    ctx.span?.setStatus({ code: 2, message: ctx.error?.message })
    ctx.span?.end()
  }
})

// Express App
const express = require('express')
const app = express()

// Middleware
app.use(express.json())

app.use(function requestLogger (req, res, next) {
  console.log(`${req.method} ${req.url}`)
  next()
})

app.use(function addRequestId (req, res, next) {
  req.requestId = Math.random().toString(36).slice(2, 10)
  res.setHeader('x-request-id', req.requestId)
  next()
})

// Routes
app.get('/users/:id', function validateId (req, res, next) {
  if (isNaN(req.params.id)) {
    return res.status(400).json({ error: 'id must be a number' })
  }
  next()
}, function loadUser (req, res, next) {
  req.user = { id: req.params.id, name: 'Test User' }
  next()
}, function getUser (req, res) {
  res.json({ ...req.user, requestId: req.requestId })
})

This produces a waterfall view looking like this:

Note: The three spans that look the same at the bottom actually represent the three handlers executing for that route, it is up to the subscriber to decide which one is the route handler and how to name the other ones since it has access to layer.

Is tracing channels experimental?

We had this come up a few times during the initiative of pushing the adoption of dc across the ecosystem, most recently this was brought up and responded to by @Qard here:

Automattic/mongoose#16105 (comment)

Yes, it has mainly been left as experimental due to a lack of adoption, but also as a signal that we may make improvements as we learn from uses of it. That learning has largely already been absorbed and applied via recent PRs at this point. There are several improvements incoming, but none of those change the existing code, which will be left as-is, they add some additional systems to handle some more specific cases such as async iterators.

The TracingChannel API will be left as-is, likely forever at this point as it is very important to us to retain compatibility essentially indefinitely, because APM vendors have many customers using old Node.js versions even well beyond the EOL of Node.js itself [...]

Related work

• This builds on Use diagnostics_channel in layers #96 but uses tracing channels
• Related to feat: add diagnostics_channel support for app initialization expressjs/express#7041
• Does the same as feat: add diagnostic channels for request lifecycle expressjs/express#7171 but uses tracing channels
• Closes Support diagnostic channels expressjs/express#6353

Caveats

• The channel name right now is pillarjs.router.request, in the original PR it was request but I think channel names need to have a relationship with the module name otherwise we risk duplication in the ecosystem.
• Depending on the exact version of Node 18 certain behaviors are expected in this PR:
  • Node versions without hasSubscribers will assume that there is a subscriber and will execute the traced path (with context object allocation). Multiple libraries like redis and mysql2 accepted this limitation.
  • Node versions without tracingChannel support will just not offer tracing of any kind. The fallback here is NOOP.
• It is up to APMs to represent the onion shape with nested handlers or flatten certain spans which is relatively easy if APMs have their own ALS context propagation strategy.
• Tests running a node version that do not have tracingChannel will be skipped via describe.skip.

[Copilot]

Pull request overview

Adds request-layer lifecycle instrumentation using Node.js diagnostics_channel TracingChannel, emitting structured events for middleware, route handlers, and error handlers to support APM-style tracing with context propagation.

Changes:

• Add express:request TracingChannel integration in Layer request/error handling.
• Introduce shared invocation helper to wrap handler execution and promise handling with tracing hooks.
• Add a new test suite validating emitted events, context shape, and basic ordering/behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
lib/layer.js	Adds TracingChannel wiring and refactors handler invocation through a tracing-aware helper.
test/tracing.js	New mocha tests covering tracing context, handler types, and lifecycle event expectations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Qard]

Generally LGTM. I think we might want to validate if handled errors are still captured on the span prior to the handler though.

[Qard]

Should this one capture the error even though it is considered handled overall?

[logaretm]

Not entirely sure, APMs will usually have a global error handler or whatever to catch them so having them traced might be an overkill and may duplicate error emissions for those who don't dedup them.

WDYT?

[Qard]

But does it having a handler here result in the error not being present on the span for the layer that lead to the error handler? Whatever the behaviour is, I think it would be useful to validate it more explicitly in this particular test. We should have a clear model of what errors happen where.

[logaretm]

Thanks! I added more tests to pin down various behaviors. Here is how errors flow right now:

No error handler (no app.use):

• Route throws sync -> error channel publishes for the route layer. Router responds 500.
• Route promise rejects -> error channel publishes for the route layer. Router responds 500.

With error handler:

• Route throws sync -> error channel publishes for the route, handler runs, nothing on the error channel for the handler.
• Route rejects async -> same, error channel publishes for the route, handler clean.

With error handler that also throws/rejects:

• Route throws sync, handler throws -> error channel publishes twice, once per layer. Two legitimately failed executions.
• Route rejects async, handler throws -> same.

For an APM: both throw cases produce two error events on the channel — one with ctx.layer pointing at the route, one at the error handler. Each event targets its own span, so the APM just marks both spans as failed.

In the case for next(err) then the child error handler span will be marked, not the parent one. Which is fine I suppose since it technically recovered as you've said.

Tests in 1ad1b9c.

[Qard]

Hmm...this handling seems a bit incorrect. If an error is unhandled it should definitely be reported. And even if it is handled we should still probably be reporting it so we can capture cases like a too permissive error handler silencing legitimate failures.

I think probably what we should be doing is always capturing the error and just adding some handled flag to the context to mark when it has been handled somehow. That way we can still report it with the detail that the application thinks it has been handled while still allowing human judgement to look at the span and see if that assessment actually makes sense.

[Qard]

This seems problematic. Unhandled errors should definitely always be observable.

[logaretm]

Okay agreed, this could've become a real gap and it's always easier to dedup rather than miss events.

I pushed a fix to that in 90ed7e2 and adjusted the tests to pin that behavior.

[Qard]

I think it's in a reasonable state now. Might be good to also get a review from @rochdev or @Flarna for some more perspectives from other APM vendors. The specific error semantics I think might be a bit up to interpretation how they should be handled, but I think this shape should generally provide the context that is needed for the different ways we think about route errors. 🤔

[Flarna]

Why is express:request channel used here?
I think it would be better to use a dedicated channel for router to allow consumers to decide on what to receive and how to handle it.
I think router component can be also used without express.

[logaretm]

Good point, I don't think it has to be express prefixed but it needs a unique prefix for sure which is why I wanted some opinions on what it should be. My concern is router:request is too generic and can easily conflict with other names in the ecosystem.

I had this in the PR description:

The channel name right now is express:request, in the original PR it was request but I think channel names need to have a relationship with the module name otherwise we risk duplication in the ecosystem.

WDYT? Maybe pillarjs:router:request?

[Flarna]

There seems to be a guideline.
So I guess the string to use would be pillarjs.router.request as tracing: prefix and event name postfix is added by dc.tracingChannel().

[logaretm]

Doesn't seem like there is a consistent standard, docs do suggest that but others just went with lib:something:action formats.

Anyways I have no strong opinions here and I think as long as the name is unique enough, it is arbitrary. I renamed it to your suggestion 👍

[bjohansebas]

I think it should just be router.request and not pillarjs.router.request. I say this because pillarjs is more like the organization name and that could change, and we might move this package to the expressjs organization (not saying we’ll do it now, but it could happen, so we shouldn’t be tied to the GitHub organization name).

[bjohansebas]

Okay, I understand the concern. In that case, instead of using pillarjs, it’s better to use express, since it’s officially under the expressjs project.

How does Fastify handle it?

[logaretm]

So should we go for express.request or express.router.request? To me its arbitrary but perhaps you have a strong opinion?

Fastify uses fastify.request.handler as seen here.

[Flarna]

I think express.request is a bit too generic. There might be other parts in express describing the request lifecycle where the router component is not included.
e.g. express.request.router or express.router.request - tending towards express.request.router like fastify and hierarchic ordering.

[logaretm]

Went with express.router.request since it describes the action better "Express's router is handling a request".

But happy to change to whatever that is acceptable to get this through.

[Flarna]

The actual data published by the tracing channel is the Layer instance, req and res objects.

If I understand the router module correct layer is an internal object which gets exposed here. Is there any guarantee that the relevant fields on that object stay intact? In fact they become part of the public interface here.

Also I don't find in the docs here that req.route is a public property. Is it safe for consumers to rely on that as it is seen part of public API and following Semver regarding changes?

In general I would love some section in the docs describing the diag channel and it's emitted data.

[logaretm]

Is it safe for consumers to rely on that as it is seen part of public API and following Semver regarding changes?

I can't answer that as it usually up to maintainers to decide this but I would say it does imply a public API contract and should be subject to smever.

As for the req.route, I'm not sure the AI created a snippet with it and it worked 🤷‍♂️

The layer object is strictly private today so this woulda actually expose it publicly.

In general I would love some section in the docs describing the diag channel and it's emitted data.

I think we have 3 options here:

• Define a curated list of properties and expose those instead as public API.
• Layer objects become a public API.
• Continue to emit layer, but document a stable partial of it.

I think that would be up to the maintainers to decide, but all options work for us.

[Qard]

I generally suggest that event contents be considered public API surface, but I would not say that is mandatory as it is very common that diagnostics_channel events would be observing what would otherwise be internal implementation details. I would ask though that any exposure of internals be explicitly called out as internal to inform consumers that they should consume those events with defensive accesses to handle cases where shapes may change.

I don't fully think calling it public API surface is quite the right characterization of the intent.

[bjohansebas]

Also I don't find in the docs here that req.route is a public property. Is it safe for consumers to rely on that as it is seen part of public API and following Semver regarding changes?

Yes, it’s part of the public API (see

router/index.js

Line 281 in cb2a5fc

req.route = route

), so it should follow semver. That means anyone can depend on it. There might not be documentation (which I’ll add to my TODO once I’m back on the project, in case no one else has done it or if it doesn’t exist yet, since I don’t have time to verify it right now).

[bjohansebas]

Please also add documentation about this in the README.

[logaretm]

@bjohansebas I added a section on diagnostic channels and their payload, and marked layer as internal impl.

[bjohansebas]

Good job! I noticed a few details, and I'll open a PR to update the documentation on the website

edit: Note that the refactors I mentioned don't fix the bugs I found.

[bjohansebas]

Since we support Node 18.0.0, users on those versions will get true, because (undefined !== false) === true.

https://nodejs.org/api/diagnostics_channel.html#diagnostics_channeltracingchannelnameorchannels

[logaretm]

Yep this is a trade off I suppose. I think we should then check for all channels own subscribers in that case.

[bjohansebas]

If someone did something like this, I know it's a very edge case, but it's still something we should fix since it's allowed by the way the router is designed:

router.get('/via-throw', function viaThrow (req, res, next) {
  throw 'route' // eslint-disable-line no-throw-literal
})
router.get('/via-throw', (req, res) => res.end('second route ran'))

the error event would still be emitted, so it wouldn't be limited to just next(err). Even so, the request would still end up returning a 200.

[logaretm]

I added an exception for these routing signals, should be ignored now.

[bjohansebas]

Okay, we have a small bug here. When there are mounted sub-routers, if an error occurs, the parent router ends up publishing a duplicate error event. For example:

const http = require('node:http')
const dc = require('node:diagnostics_channel')
const Router = require('/home/bjohansebas/Dev/oss/pillarjs/router')

const errorEvents = []

dc.tracingChannel('express.router.request').subscribe({
  error (ctx) {
    errorEvents.push({
      layer: ctx.layer && ctx.layer.name,
      handled: !!ctx.handled,
      message: ctx.error && ctx.error.message
    })
  }
})

function report (title) {
  console.log('\n%s', title)
  console.log('  error events published: %d %s', errorEvents.length,
    errorEvents.length === 1 ? '(OK)' : '(BUG: should be exactly 1, on the origin layer)')
  for (const e of errorEvents) {
    console.log('    layer=%s handled=%s message=%j', e.layer, e.handled, e.message)
  }
  errorEvents.length = 0
}

async function run (router, path) {
  const server = http.createServer(function (req, res) {
    router(req, res, function (err) {
      res.statusCode = err ? 500 : 404
      res.end(String(err || 'not found'))
    })
  })
  await new Promise(r => server.listen(0, r))
  await fetch('http://localhost:' + server.address().port + path)
  await new Promise(r => setImmediate(r))
  server.close()
}

async function main () {
  // Scenario A: mounted sub-router — a single next(err) in the inner handler,
  // recovered by an outer error handler.
  {
    const outer = new Router()
    const nested = new Router()

    nested.get('/bar', function innerHandler (req, res, next) {
      next(new Error('boom'))
    })

    outer.use('/foo', nested)

    outer.use(function recover (err, req, res, next) { // eslint-disable-line no-unused-vars
      res.statusCode = 500
      res.end(err.message)
    })

    await run(outer, '/foo/bar')
    report('Scenario A: nested router (1 mount level)')
  }

  // Scenario A2: two mount levels — the error is republished on EVERY ancestor.
  {
    const outer = new Router()
    const mid = new Router()
    const deep = new Router()

    deep.get('/baz', function deepHandler (req, res, next) {
      next(new Error('boom'))
    })

    mid.use('/bar', deep)
    outer.use('/foo', mid)

    outer.use(function recover (err, req, res, next) { // eslint-disable-line no-unused-vars
      res.statusCode = 500
      res.end(err.message)
    })

    await run(outer, '/foo/bar/baz')
    report('Scenario A2: nested router (2 mount levels)')
  }

  // Scenario B: error handler that forwards the error with next(err).
  {
    const router = new Router()

    router.get('/fail', function origin (req, res, next) {
      next(new Error('boom'))
    })

    router.use(function forwardingHandler (err, req, res, next) {
      next(err) // does not handle it, delegates to the next one
    })

    router.use(function recover (err, req, res, next) { // eslint-disable-line no-unused-vars
      res.statusCode = 500
      res.end(err.message)
    })

    await run(router, '/fail')
    report('Scenario B: error handler forwarding with next(err)')
  }
}

main()

In principle, the error event should only be published once.

[logaretm]

That's a good catch. I will dedup the errors published in relation to the request object.

[bjohansebas]

LGTM. It looks like tracePromise cuts performance roughly in half when tracing is enabled. I'm not sure if that's expected behavior for an APM. We could probably improve performance a bit by using .start.runStores, although that would be semantically different.

[bjohansebas]

I think we should rename it to something like errorHandler, maybe?

Suggested change

	- `handled`: `true` when the layer is an error-handling middleware (4-arg signature)
	- `errorHandled`: `true` when the layer is an error-handling middleware (4-arg signature)

[logaretm]

That's fair, the clearer the better I suppose. Maybe isErrorHandler or errorHandler actually? since it doesn't mean that the error is handled, just that the handler is an error handler.

[logaretm]

We could probably improve performance a bit by using .start.runStores, although that would be semantically different.

Yea that would mean it doesn't follow the tracing channels regular flow which can be important for APMs since it has to be predictable. tracePromise should essentially be a syntax sugar over start.runStores.

Let me see if we can benchmark this.

[bjohansebas]

I'm testing on my local machine, which isn't the most stable environment for benchmarks, but I couldn't find much else to optimize. I also think Fastify and others pay the same cost when using tracePromise, so it seems more like a Node.js overhead than something we can really improve on our side

[logaretm]

So I did a test and you are right it is 1.9x faster for me, but only on the sync handlers case. Async handlers are more or less the same.

We could do what we did with GraphQL and strap the whole 5 events with runStores and manual publishes. Let me see if that gives us a win in sync case and passes the context propagation tests, it should.

EDIT: Just did, for sync handlers it will effectively follow traceSync behavior, for non async handlers it will follow the same semantics as tracePromise. This regains 1.9x speed for those sync handlers on my machine.

[bjohansebas]

SGTM. @Qard @Flarna , since you have more context on this as well, do you think this is a good approach?