[ISSUE-378][HugePartition][Part-1] Record every partition data size for one app #458
Conversation
| /** | ||
| * shuffleId -> partitionId -> partition shuffle data size | ||
| */ | ||
| private Map<Integer, Map<Integer, Long>> partitionDataSizes; |
There was a problem hiding this comment.
It will be added by the ConcurrentHashMap.computeIfPresent, this is a thread safe operation. Right?
There was a problem hiding this comment.
What will it happen if two threads add the same partition?
There was a problem hiding this comment.
This will guarded by computeIfPresent to keep thread safe when having multiple threads.
There was a problem hiding this comment.
You can see the code of computeIfpresent. computeIfpresent don't guard the atomicity. Only the one partition will be added.
There was a problem hiding this comment.
There was a problem hiding this comment.
OK, my mistake. I just see the default method of Map.
Codecov Report
@@ Coverage Diff @@
## master #458 +/- ##
============================================
+ Coverage 58.67% 58.74% +0.07%
- Complexity 1655 1664 +9
============================================
Files 199 199
Lines 11214 11236 +22
Branches 997 999 +2
============================================
+ Hits 6580 6601 +21
- Misses 4242 4243 +1
Partials 392 392
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Could you modify the title? What's |
n means several subtasks. In the next PR, it will named as 2/n. |
So n is 3 or 4? If we can't ensure what n is. Could we use |
Yes. It' OK for me. Could you help modify the title? |
jerqi
changed the title
OK. |
|
Thanks @jerqi |
What changes were proposed in this pull request?
Record every partition data size for one app
Why are the changes needed?
This is a subtask for #378
Does this PR introduce any user-facing change?
No
How was this patch tested?