Skip to content

Emitter fails to send metrics to Monitor after ~300 messages #10

Description

@sunefred

This bug appears as 400: BadRequest after about 300 messages has been sent to Azure Monitor. This is the error message received from the Azure Monitor metrics endpoint:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML>

<HEAD>
  <TITLE>Bad Request</TITLE>
  <META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii">
</HEAD>

<BODY>
  <h2>Bad Request - Request Too Long</h2>
  <hr>
  <p>HTTP Error 400. The size of the request headers is too long.</p>
</BODY>

</HTML>

This issue happens because the header grows for each request as application/json is added to the default request header in EmitterHelper

_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

Instead we should create a new HttpRequestMessage for every request, or optionally clear the Accept header. This would be the updated code for SendCustomMetricAsync

public async Task<HttpResponseMessage> SendCustomMetricAsync(
    string? region, string? resourceId, EmitterSchema metricToSend,
    CancellationToken cancellationToken = default)
{
    if ((region != null) && (resourceId != null))
    {
        var record = await _TokenStore.RefreshAzureMonitorCredentialOnDemandAsync(cancellationToken);

        string uri = $"https://{region}.monitoring.azure.com{resourceId}/metrics";
        string jsonString = JsonSerializer.Serialize(metricToSend, _jsonOptions);

        using var request = new HttpRequestMessage(HttpMethod.Post, uri)
        {
            Content = new StringContent(jsonString, Encoding.UTF8, "application/json")
        };

        request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", record.Token);
        request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

        _logger.LogInformation("SendCustomMetric:{uri} with payload:{payload}", uri, jsonString);

        return await _httpClient.SendAsync(request, cancellationToken);
    }

    return new HttpResponseMessage(HttpStatusCode.LengthRequired);
}

And similarly there needs to be an update for GetAllConsumerGroups.

❓Is there an appetite for fixing issues in Azure-Samples, specifically this sample? If so, I could open pull requests for fixes we have made in our fork.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions