The query frontend logs helpful messages for every API response like this:
level.Info(logger).Log(
"msg", "query range response",
"tenant", tenantID,
"query", req.Query,
"range_nanos", req.End-req.Start,
"max_series", req.MaxSeries,
"duration_seconds", durationSeconds,
"request_throughput", float64(resp.Metrics.InspectedBytes)/durationSeconds,
"total_requests", resp.Metrics.TotalJobs,
"total_blockBytes", resp.Metrics.TotalBlockBytes,
"total_blocks", resp.Metrics.TotalBlocks,
"completed_requests", resp.Metrics.CompletedJobs,
"inspected_bytes", resp.Metrics.InspectedBytes,
"inspected_traces", resp.Metrics.InspectedTraces,
"inspected_spans", resp.Metrics.InspectedSpans,
"partial_status", resp.Status,
"partial_message", resp.Message,
"num_response_series", len(resp.Series),
"error", err)
We need to include trace ID, so we can easily navigate from the log to the trace for that request. This is probably a straight-forward fix because we already have traces along the read path, but I'd like to double check - was this a simple miss, or is there a reason it wasn't included in the first place?