Ensure DataModel children can be correctly cleaned up

[philippjfr]

Fixes #3581
Fixes #5920

[codecov]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (16f876a) 41.14% compared to head (bec2436) 84.15%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
panel/reactive.py	85.00%	3 Missing ⚠️
panel/io/datamodel.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #5924       +/-   ##
===========================================
+ Coverage   41.14%   84.15%   +43.01%     
===========================================
  Files         291      291               
  Lines       42793    42807       +14     
===========================================
+ Hits        17606    36024    +18418     
+ Misses      25187     6783    -18404     

Flag	Coverage Δ
ui-tests	40.92% <12.50%> (+<0.01%)	⬆️
unitexamples-tests	72.43% <79.16%> (+49.60%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.