I'm very ignorant in the workqueue area, but should we check the NumRequeues too?

Good question - I added a comment here to clarify that this was intentional. I think we'd want to wait indefinitely for the informer cache to update but that may be naive. @lavalamp do you have any tips here?

If the informer cache can't update you'll have bigger problems. That kind of problem should be caught by monitoring the metrics.

@lavalamp What kind of metric could I expose here? If I simply increment a counter every time we run into this error I think we'd end up with a lot of noise. Maybe a metric that tracks every time we go past max retries and give up for a Service?

Update - I've removed this extra check and this error will not get any kind of special treatment.

If the only possible cause is the informer not having received an update, that will be reflected in existing metrics (it will be missing all updates in that case). If there's any chance of another kind of error, my first thought is to make two gauge metrics, one that counts the number of items in this condition, and one that adds up how many seconds those items have been in the condition. That way you can tell the difference between 1 persistently bad item and having a different item in this condition every time (which is not a problem).

I think right now the other potential cause of an issue here would be someone manually setting an EndpointSlice generation lower than a previous value. As long as we can seal that up with a separate PR to prevent that in the strategy logic, I don't think we'll need much in the way of extra metrics.

For the two gauge metrics would you start a separate goroutine that did these calculations at some kind of fixed interval for all Services?

I've been thinking about adding a new metric that would track the number of syncs the controller has performed along with the outcome as a label with 3 potential values: success, staleCache, and error. That would not be as helpful in identifying outliers but would provide some broad insight into the health of the controller.

Since I'm interested in attempting to backport this, I think adding a metric directly to this PR would complicate that, so maybe this can be done as a follow up.