Add more configuration flexibility to Egress for user

[jainpulkit22]

Added a StrictEnforcement field to the Egress spec. If this field is set to flase then if the egress node is not available due to any reason then the packet transfer or we can say the traffic will go via normal Node SNAT, while in the second case where this field is set to true, and if egress node is not available packet won't go out and traffic will be stuck
because of unavailability of egress node.

Fixes #6988.

[antrea-bot]

Can one of the admins verify this patch?

[jainpulkit22]

Points to note:

1. In case of dynamic egress, if there is no node available then egress IP is also not assigned to the egress, and so no flows are installed and traffic goes through normal Node SNAT, so its not possible to add configurable parameter for dynamic egress because in case of no node available we don't have any egress, and we don't have any deciding condition on basis of which we can drop the traffic if we want to enable strict enforcement for dynamic egress.
2. In case of static egress what happens is the Egress has an IP, but node is not guaranteed so we have two choices now that when egress node is not available for static egress we can either allow the traffic to go through normal Node SNAT(just similar to the working of dynamic egress), or we can drop the traffic.

My current implementation gives user the flexibility to decide whether to allow the traffic(best effort), or not(strict enforcement of egress policy).

But In ideal case won't this be too confusing for the end user that static egress is configurable and dynamic is not, so should we keep the behaviour of both the egress consistent and make sure that when there is no node available for static egress then instead of dropping the traffic/packet we should allow the traffic to pass through the normal Node SNAT, same as in the case of dynamic egress.

what's your opinion on this @tnqn @antoninbas

[jainpulkit22]

Points to note:

1. In case of dynamic egress, if there is no node available then egress IP is also not assigned to the egress, and so no flows are installed and traffic goes through normal Node SNAT, so its not possible to add configurable parameter for dynamic egress because in case of no node available we don't have any egress, and we don't have any deciding condition on basis of which we can drop the traffic if we want to enable strict enforcement for dynamic egress.
2. In case of static egress what happens is the Egress has an IP, but node is not guaranteed so we have two choices now that when egress node is not available for static egress we can either allow the traffic to go through normal Node SNAT(just similar to the working of dynamic egress), or we can drop the traffic.

My current implementation gives user the flexibility to decide whether to allow the traffic(best effort), or not(strict enforcement of egress policy).

But In ideal case won't this be too confusing for the end user that static egress is configurable and dynamic is not, so should we keep the behaviour of both the egress consistent and make sure that when there is no node available for static egress then instead of dropping the traffic/packet we should allow the traffic to pass through the normal Node SNAT, same as in the case of dynamic egress.

what's your opinion on this @tnqn @antoninbas