Dynamically Routing Traces to Customer-Specific App Insights

When building a SaaS offering publishing platform that helps customers publish azure marketplace offers, we faced an interesting observability challenge: how do you maintain separate telemetry streams for both your platform and each customer's application?

The Problem

Our platform automatically provisions Azure Application Insights instances for each marketplace offer. Customers can send telemetry to their dedicated instance, but we also need to route our internal billing and management traces to the same destination. The tricky part is to dynamically determine and route the relevant traces to customer app insights based on billing event context.

The Solution: Dynamic Telemetry Routing

We built a factory pattern that creates subscription-specific OpenTelemetry exporters on-demand:

from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter

class TelemetryExporterFactory:
    def __init__(self):
        self.key_vault_service = KeyVaultService(settings.keyvault_url)
        self._connection_string_cache = {}
        self._connection_string_lock = asyncio.Lock()

    async def get_trace_exporter(self, mp_sub_id: str, db: AsyncSession):
        connection_string = await self._get_connection_string(
            mp_sub_id=mp_sub_id, db=db
        )
        return AzureMonitorTraceExporter(connection_string=connection_string)

    async def _get_connection_string(self, mp_sub_id: str, db: AsyncSession):
        if mp_sub_id in self._connection_string_cache:
            return self._connection_string_cache[mp_sub_id]

        async with self._connection_string_lock:
            ... # find and return the connection string for the specific offer given a specific subscriber

While also setting up our own observability stack:

def setup_observability(app: FastAPI | None = None):
    # Set up platform-wide observability for our SaaS platform
    LoggingInstrumentor().instrument(set_logging_format=True)
    configure_azure_monitor(credential=DefaultAzureCredential())

    tracer = trace.get_tracer(__name__)
    if app:
        FastAPIInstrumentor.instrument_app(app)

Clean Usage with Context Managers

The real magic happens with our convenience wrapper that makes subscription-specific tracing feel natural:

def create_subscription_span(mp_sub_id: str, name: str, factory=None, db_session=None):
    @asynccontextmanager
    async def _span_context():
        with get_tracer().start_as_current_span(name) as span:
            yield span
            if factory and db_session and span:
                await factory.export_span(mp_sub_id=mp_sub_id, span=span, db_session=db_session)

    return _span_context

# Usage in billing operations
async with create_subscription_span(
    subscription_id,
    "billing.process_usage",
    telemetry_factory,
    db_session
) as span:
    span.set_attribute("usage.quantity", quantity)
    span.set_attribute("usage.dimension_id", dimension_id)
    await process_billing_event(subscription_id, usage_data)

Why This Matters

Each customer gets their own observability stack, completely isolated
We can separate logging and tracing for our own observability stack, and add span events & attributes for information relevant to our customers
Customer traces and platform traces appear in the same App Insights instance and can be correlated
No configuration needed - the system automatically routes based on subscription