Fix TestReconcileBatchJob

[forsaken628]

What this PR does / why we need it:
Fix unstable tests
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
#1649

Checklist:

• Docs included if any changes are user facing

[tenzen-y]

@forsaken628 Thank you for this contribution.
Could you explain the reason why the flakiness happened and what you fixed the root cause in the PR description?

[forsaken628]

According to the previous logic，call GetTrialObservationLog will return observationLogAvailable once and then observationLogUnavailable multiple times.

I investigated the cause of the failure in the ci environment: in test 2, which should have returned observationLogAvailable, it actually returned observationLogUnavailable, indicating that GetTrialObservationLog was accidentally called once somewhere.

Instead of locating where the accident occurred, I rewrote mockManagerClient to always return the same reply before calling DeleteTrialObservationLog, which I think is consistent with the semantics of ManagerClient.

[tenzen-y]

According to the previous logic，call GetTrialObservationLog will return observationLogAvailable once and then observationLogUnavailable multiple times.

I investigated the cause of the failure in the ci environment: in test 2, which should have returned observationLogAvailable, it actually returned observationLogUnavailable, indicating that GetTrialObservationLog was accidentally called once somewhere.

Instead of locating where the accident occurred, I rewrote mockManagerClient to always return the same reply before calling DeleteTrialObservationLog, which I think is consistent with the semantics of ManagerClient.

Thank you for clarifying the root cause. It seems that you split out multiple test case into dedicated case based on the result of the GetTrialObservationLog. That sounds reasonable.

[tenzen-y]

TBH, we should reimplement these tests as ginkgo BDD style, but that is out of scope of this PR.
So, I leave such refactoring in the future.

[tenzen-y]

Suggested change

	mockMC.msg = observationLogUnavailable
	mockManagerClient.EXPECT().GetTrialObservationLog(gomock.Any()).Return(observationLogAvailable, nil).AnyTimes()

Instead of this original mock client, can we reuse the autogenerated mock clients like this?

[forsaken628]

We need WithOverridableExpectations here, but github.com/golang/mock/mock does not support this feature.
Or "Each test should create a new Controller".
In my opinion, gomock is not flexible enough, and it complicates a simple thing for this scenario.

[forsaken628]

Finally, I found another way.

[tenzen-y]

@forsaken628 I opened the issue to replace gomock with go.uber.com/gomock here: #2356

As a separate PR, could you work on the issue?

[tenzen-y]

/lgtm
/approve

Thank you for the great contribution!

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

Thank you for this amazing contribution @forsaken628!
I really hope it will improve our unstable tests.