Add option to override driver + executor memory limit using webhook #2383
Conversation
Signed-off-by: Netanel Levine <[email protected]>
|
Hi there, I'm from @netanellevine team, this feature was proven to be very important to us, as instead of defining for each app the max memory amount used, we were able to define a much lower number and save both time and money. |
|
@netanellevine @bnetzi I am glad you were able to take that on. It seems like a very useful enhancement. I am wondering if you would need UT as well. I would be happy to help on that (it would be my first contribution, so let me know if I am a distraction ). |
|
Hi @danielrsfreitas we will be glad if you can add UT, we didn't got to it (although we did build it and tested it in our env) |
|
|
||
| // Apply the memory limit to the container's resources | ||
| logger.Info(fmt.Sprintf("Adding memory limit %s to container in pod %s", *memoryLimit, pod.Name)) | ||
| pod.Spec.Containers[i].Resources.Limits[corev1.ResourceMemory] = limitQuantity |
There was a problem hiding this comment.
I was only able to make my unit tests work after initializing the pod.Spec.Containers.Resources.Limits, like this
pod.Spec.Containers[i].Resources.Limits = corev1.ResourceList{}
pod.Spec.Containers[i].Resources.Limits[corev1.ResourceMemory] = limitQuantity
There was a problem hiding this comment.
Hi @danielrsfreitas , I've added a fix to verify that pod.Spec.Containers.Resources.Limits is initialized. I also ran the tests locally, and they passed.
Signed-off-by: Netanel Levine <[email protected]>
|
Hi @jacobsalway @vara-bonthu - could you please review this feature? |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: danielrsfreitas The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| // MemoryLimit is overriding the driver pod memory limit. | ||
| // By default, the memory limit and request are equal - use carefully. | ||
| // +optional | ||
| MemoryLimit *string `json:"memoryLimit,omitempty"` |
There was a problem hiding this comment.
I think it is better to define MemoryLimit in SparkPodSpec as it appears in both driver/executor pod.
There was a problem hiding this comment.
FYI @bnetzi let me know if you need assistance addressing those comments. :)
There was a problem hiding this comment.
Hi @danielrsfreitas thanks!
We will be able to handle fixes only in 1-2 weeks.
As for the comment itself - I think it's better to have it in pod spec but we haven't had the time to see if it makes sense and if there is anything preventing it.
| func convertJavaMemoryStringToK8sMemoryString(memory string) string { | ||
| if strings.HasSuffix(memory, "Gi") || strings.HasSuffix(memory, "Mi") { | ||
| return memory | ||
| } | ||
| // Convert the memory string from 'g' to 'Gi' and from 'm' to 'Mi. | ||
| if strings.HasSuffix(memory, "g") || strings.HasSuffix(memory, "m") { | ||
| return strings.ToUpper(memory) + "i" | ||
| } | ||
|
|
||
| return memory | ||
|
|
||
| } |
There was a problem hiding this comment.
We can create a new util function from this and place it in file pkg/util/util.go, it would be better if more units (e.g. k, kb, m, mb, g, gb, t, tb, p, pb) can be converted to k8s memory unit.
Purpose of this PR
Add an option to override the driver and executor memory limit as by default spark uses equal limit and request.
It is Done by webhook as the pod template does not support overriding the memory limit.
Change Category
Rationale
For many use cases, it makes sense to share memory between pods as they don't all peak simultaneously. Limit allows better resources allocation
Checklist