level=ERROR msg="error received after stop sequence was engaged" err="leader election lost"

[mxey]

When running a controller manager with LeaderElection enabled, shutting down the manager will obviously release the leader lock. However, this seems to still trigger a “leader election lost” error:

level=ERROR msg="error received after stop sequence was engaged" err="leader election lost"

On its own, this is just a nitpick, but the problem is in combination with using the t.Output() writer in your test functions. Tests are not allowed to keep writing to it after the test has finished. You will either get a panic, or might even get a data race detected if you run the test with the race detector.

The problem here is that the manager keeps doing something even after its Start method has already returned. The log is a symptom of that, but IMO the manager should be completely gone once Start returns.

[mxey]

I think the culprit is in engageStopProcedure where this goroutine is started but nothing waits for it to finish:

	go func() {
		for {
			// Closing in the for loop is required to avoid race conditions between
			// the closure of all internal procedures and making sure to have a reader off the error channel.
			closeOnce.Do(func() {
				// Cancel the internal stop channel and wait for the procedures to stop and complete.
				close(cm.internalProceduresStop)
				cm.internalCancel()
			})
			select {
			case err := <-cm.errChan:
				if !errors.Is(err, context.Canceled) {
					cm.logger.Error(err, "error received after stop sequence was engaged")
				}
			case <-stopComplete:
				return
			}
		}
	}()

[liketosweep]

I'd like to investigate this if no one is working on it