Debugging Model Performance on 1 Continuous Target: CTBiomarkers.CalciumScoring.AbdominalAgatston

[Ahmed14974]

Previous comments:

Ahmed14974: As previously observed through preliminary modeling iterations, any and all models trained on the input images to predict the Continuous Target variable CTBiomarkers.CalciumScoring.AbdominalAgatston have not shown any degree of decent performance. This Issue will document all approaches to be considered for the purposes of tuning and improving model performance, including model iterations which have already been attempted

echen4096: I tested a ResNet model with CoordConv layers and pretrained weights (commit). However, model performances remain poor.

Results:

• Original Resnet Model

mse: 0.013112680986523628,
r_squared: -0.013185453842119443,
explained_variance: -0.010275483131408691

• Resnet Model with CoordConv

mse: 0.01319037564098835,
r_squared: -0.019188805176170165,
explained_variance: -0.004194855690002441

[Ahmed14974]

Experiments Attempted for CTBiomarkers.CalciumScoring.AbdominalAgatston

Details of models attempted with architecture specifics and results are presented below. For all models, the target variable is min-max rescaled to a range of [0,1].

• ResNet-34 (Train/Val/Test Batch Size of 64)

Exp_name	View	Learning Rate	Head Attached	Sigmoid Layer Appended to Head	model only_last_layer	Image Transformations	Num Epochs	Test MSE	Test R-squared	Test Explained Variance Score
mahmedc_resnet34_1e-3_two_view	two_view	1e-3	No	No	FALSE	legacy	10	0.013425091	-0.0373247	-0.015828609
mahmedc_resnet34_1e-3_one_view	one_view	1e-3	No	No	FALSE	legacy	10	0.013014367	-0.0055890	-0.000684977
mahmedc_resnet34_1e-3_one_view_head_model	one_view	1e-3	Yes	No	FALSE	legacy	10	0.013009196	-0.0051894	-0.004006982
mahmedc_resnet34_1e-3_one_view_head_model_sigmoid	one_view	1e-3	Yes	Yes	FALSE	legacy	10	0.013236634	-0.0227629	-0.000447869
mahmedc_resnet34_1e-3_one_view_head_model_sigmoid_onlylast	one_view	1e-3	Yes	Yes	TRUE	legacy	10	0.013681340	-0.0571244	-0.024955988
mahmedc_resnet34_1e-3_one_view_head_model_sigmoid_onlylast_transforms	one_view	1e-3	Yes	Yes	TRUE	alligned with ResNet pre-training	10	0.013806610	-0.0668038	-0.013790250
mahmedc_resnet34_1e-4_one_view_head_model_sigmoid_onlylast_transforms_50	one_view	1e-4	Yes	Yes	TRUE	alligned with ResNet pre-training	50	0.013779610	-0.0647176	-0.002306342
mahmedc_resnet34_1e-5_one_view_head_model_sigmoid_onlylast_transforms_50	one_view	1e-5	Yes	Yes	TRUE	alligned with ResNet pre-training	50	0.049071338	-2.7916246	-0.243152857

Training and Validation Loss Curves (MSE) for the model with lowest Test MSE from above

• ResNet-101 (Train/Val/Test Batch Size of 16)

Exp_name	View	Learning Rate	Head Attached	Sigmoid Layer Appended to Head	model only_last_layer	Image Transformations	Num Epochs	Test MSE	Test R-squared	Test Explained Variance Score
mahmedc_resnet101_1e-4_one_view_head_model_sigmoid_transforms_50_batch16	one_view	1e-4	Yes	Yes	FALSE	alligned with ResNet pre-training	50	0.015156988	-0.171144077	0

Head attached is as follows (Sigmoid appended as the last layer, not present in some experiments)

class Head(nn.Module):
    def __init__(self, in_planes=1024, mid_planes=512, out_planes=35):
        super().__init__()

        self.layers = nn.Sequential(
            #AdaptiveConcatPool2d(sz=(1, 1)),
            #nn.Flatten(),
            nn.BatchNorm1d(in_planes, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
            nn.Dropout(p=0.25, inplace=False),
            nn.Linear(in_features=in_planes, out_features=mid_planes, bias=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm1d(mid_planes, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
            nn.Dropout(p=0.5, inplace=False),
            nn.Linear(in_features=mid_planes, out_features=out_planes, bias=True),
            nn.Sigmoid())

    def forward(self, x):
        return self.layers(x)

Image Transforms before and after alignment with ResNet pre-training are as follows

## Old Transforms
train_transform = transforms.Compose([
        transforms.RandomHorizontalFlip(p=0.2),
        transforms.RandomApply([
            transforms.RandomAffine(10),
            transforms.RandomResizedCrop(256, scale=(1.0, 1.1), ratio=(0.75, 1.33)),
            transforms.ColorJitter(brightness=0.2, contrast=0.2)
        ], p=0.75),
        transforms.RandomPerspective(distortion_scale=0.2, p=0.75),
        transforms.ToTensor(),
        transforms.Normalize((0.55001191,), (0.18854326,))
    ])

## Aligned Transforms
train_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.Grayscale(num_output_channels=3),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])