I'm still not convinced we want to have that for cacher - this will always be equal to the number of fetched.

So I would personally remove that metric.

Please remove - it's not used anywhere

What would be the use case for the index label? I can see how these metrics can be useful per resource, but I don't think that having the index used to filter the list is useful to the consumers of these metrics. Also, without the actual value used when matching the index, it is difficult to come up with actual conclusions since based on the value, different objects will be fetched and returned.

The use case is to understand the impact of indexing on the costs of handling LIST requests. Just knowing the index name lets us know which indices (if any) are proving helpful.

I would love to have histograms instead of counters, but that was rejected.

I have added a commit that dares to add one histogram. Perhaps that is not too much to ask?

Considering the insights you are trying to get from the index label, it makes sense to me to keep it.

This looks a bit strange to me. I mean - not the fact that you want histogram, but the fact that we have 1 histogram and N counters. It just looks very inconsistent to me.

If you really believe we need a histogram (I'm personally not convinced - I think per-time-interval delta is what we need here which we get from counter), then my question is why counters are enough in other cases.

I think that "either all or nothing" is the way to proceed.

I break usage scenarios down into two broad categories, profiling by developers and experience by real users in the wild. For the former, it is relatively easy to get the information we want from controlled experiments and count metrics. For metrics collected in the wild, there was only one traffic mix when a problem happens so it is more difficult to draw conclusions. Not completely impossible, one can compare with metrics from more normal situations. But more difficult. So I do not see it as a black-and-white issue, rather a matter of degree. And there is the contrary consideration of the volume of metrics, which is also a matter of degree. So I see our task here as balancing two matters of degree.

I am also not convinced that having histograms is meaningful here since it lacks precision compared to what we had before with counters. To me with these metrics you are interested in knowing exactly how many objects are fetched in the cache over a certain time frame based on their resource and index to measure the efficiency of indexing, which is something that you won't know from the histograms.

The histogram gives strictly more information than the two counters it replaced. Those two counters have the same content as the _sum and _count time series in the histogram, and the buckets add a rough breakdown.

For metrics collected in the wild, there was only one traffic mix when a problem happens so it is more difficult to draw conclusions. Not completely impossible, one can compare with metrics from more normal situations. But more difficult. So I do not see it as a black-and-white issue, rather a matter of degree.

Thinking more about this...
My question is - how one would one use those additional information from the histogram. They can quickly say if there were fewer larger lists or more smaller one. And what do they get from it? How to use that information?
So I don't think it's just a matter of degree - it's a matter of how to use those metrics.
So unless you have a clear case on your mind on how exactly to use that additional information (note that we already have metrics for number of particular calls, so users would already know "the average size of a request") - I'm actually against this histogram and I would revert to the previous version.

The information in the buckets gives a rough idea of how effective the indices are.

This isn't particularly useful information, because we have all the data to answer this question already.
There are exactly 4 indices:

• pods by spec.NodeName [we know how many pods on a given node is]
• secrets by metadata.Name [always exactly 1]
• configmaps by metadata.Name [always exactly 1]
• nodes by metadata.Name [always exactly 1]
  Users have no way to influence that - you need to update k8s code and recompile to be able to change that. So if someone is doing that - they can also add metrics if they really need.
  Also - the index is a label in your metric.

The buckets also give rough information on how the load is divided among resources.

resource_prefix is a label - so we have that information independently if this is histogram or not.

So I actually don't but the above argument.

Oh, interesting, I did not know those specifics about the actually existing indices.

On the second part, sorry, I botched what I was trying to say. The idea is that for a given resource, there can be a variety of observations depending on other factors (such as populations in namespaces). The buckets give some rough information about the distribution of that variety.

On the second part, sorry, I botched what I was trying to say. The idea is that for a given resource, there can be a variety of observations depending on other factors (such as populations in namespaces). The buckets give some rough information about the distribution of that variety.

Sure - but the question is how we can use that information. Let's say that instead of knowing that there were 100,000 object fetches I now that there were 100 of size 100 and 9 of size 10,000. What I can use that information for?

It can get useful in combination with other information. For one example, if I am investigating a situation with a few spikes in LIST latency for a particular resource, that information helps me narrow down my search for the cause(s). For another example, if I am trying to understand the distribution of LIST latencies for a given resource for the sake of tuning our coefficients in API Priority and Fairness, knowing the distribution of numbers fetched helps with that.

Those are somewhat hypothetical examples to me. Metrics indeed try to pin-point the approximate area of the problem, but it's really help to debug something purely on metrics. And if we pinpoint the latency of LIST requests of a particular resource in a particular point in time, we have logs (and traces in the logs) that would allow us to actually debug it.

For the tuning example - what you described sounds a bit like an attempt to reverse-engineer the expected charactestics introduced by the code from the metrics. Sure - we can do that, but is that really what we want?
Also - can we really do it if we have only partial data for it (only number of fetched objects and only for lists served from cache)? I would be much closer to buy that if we would have all 4 things being histograms [though I'm still not convinced we need that].