ULM Through DL 7(Randomized Validation Set)

In these sets of experiments, we repeated the same procedure described in the previous post with only one difference. So, please see the details in the previous post. The difference is the way that we partitioned the patches into training and validation sets. We obtained patches from 8 simulation images from Field2. Then, randomly pick 1/8 of patches and put them into validation set. The remaining patches formed the training set.

Note: Best and Worst cases are chosen by comparing the average bubble error in a patch.

Quantitative Metrics: In order to compare success of different experiments, we decide the use the followings:

Quantitative Metric1 = \[ L1Loss(z**f_0 – x**f_0) \]

Quantitative Metric2 = \[ MSELossLoss(z**f_0 – x**f_0) \]

where f_0 is a gaussian kernel with sigma =1.

Note: Both f and f_0 are in pixel coordinates.

Experiment 1: learning rate=1e-5, sigma of kernel f=1, regularizer parameter=0.01

Average quantitative metric1 is 19.645459

Average metric1 per bubble is 0.7747

Average quantitative metric2 is 0.981560

Average metric2 per bubble is 0.0384

Worst Case in terms of Per Bubble Error: Center Location(x,z) = 34.53 mm and 10.85 mm, Average Metric1 per bubble: 1.31, Average Metric2 per bubble: : 0.0756

Best Case in terms of Per Bubble Error:Center Location(x,z) = 19.75 mm and 44.37 mm , Quantitative Metric1: 0.511, Quantitative Metric2: 0.0173

Experiment 2: learning rate=1e-5, sigma of kernel f=1, regularizer parameter=0.005

Average quantitative metric1 is 20.352445

Average metric1 per bubble is 0.802987

Average quantitative metric2 is 0.985063

Average metric2 per bubble is 0.038526

Worst Case in terms of Per Bubble Error:Center Location(x,z) = 29.61 mm and 9.87 mm , Average Metric1 per bubble: : 1.235, Average Metric2 per bubble: : 0.0755

Best Case in terms of Per Bubble Error:Center Location(x,z) = 24.68 mm and 33.53 mm , Average Metric1 per bubble: : 0.538, Average Metric2 per bubble: : 0.0134

Experiment 3: learning rate=1e-5, sigma of kernel f=1.5, regularizer parameter=0.01

Average quantitative metric1 is 18.815349

Average metric1 per bubble is 0.742511

Average quantitative metric2 is 0.936104

Average metric2 per bubble is 0.036656

Worst Case in terms of Per Bubble Error:Center Location(x,z) = 2.50 mm and 13.81 mm , Average Metric1 per bubble: : 1.1237, Average Metric2 per bubble: : 0.071355

Best Case in terms of Per Bubble Error:Center Location(x,z) = 12.36 mm and 41.41 mm , Average Metric1 per bubble: : 0.522, Average Metric2 per bubble: : 0.0179

Experiment 4: learning rate=1e-5, sigma of kernel f=1.5, regularizer parameter=0.005

Average quantitative metric1 is 19.907855

Average metric1 per bubble is 0.785045

Average quantitative metric2 is 0.928157

Average metric2 per bubble is 0.036292

Worst Case in terms of Per Bubble Error:Center Location(x,z) = 2.50 mm and 17.76 mm , Average Metric1 per bubble: : 1.175, Average Metric2 per bubble: : 0.068

Best Case in terms of Per Bubble Error:Center Location(x,z) = 24.68 mm and 33.53 mm , Average Metric1 per bubble: : 0.519, Average Metric2 per bubble: : 0.0144

Experiment 5: learning rate=1e-5, sigma of kernel f=2, regularizer parameter=0.01

Average quantitative metric1 is 19.071611

Average metric1 per bubble is 0.753008

Average quantitative metric2 is 1.000624

Average metric2 per bubble is 0.039267

Worst Case in terms of Per Bubble Error:Center Location(x,z) = 2.50 mm and 13.81 mm , Average Metric1 per bubble: 1.045525, Average Metric2 per bubble: 0.065981

Best Case in terms of Per Bubble Error:Center Location(x,z) = 17.27 mm and 27.61 mm , Average Metric1 per bubble: 0.571219, Average Metric2 per bubble: 0.0237

Experiment 6: learning rate=1e-5, sigma of kernel f=2, regularizer parameter=0.005

Average quantitative metric1 is 19.766309

Average metric1 per bubble is 0.780344

Average quantitative metric2 is 0.954419

Average metric2 per bubble is 0.037464

Worst Case in terms of Per Bubble Error:Center Location(x,z) = 34.53 mm and 22.68 mm , Average Metric1 per bubble: 1.182416, Average Metric2 per bubble: 0.063596

Best Case in terms of Per Bubble Error:Center Location(x,z) = 22.21 mm and 37.47 mm , Average Metric1 per bubble: 0.561, Average Metric1 per bubble: 0.01984

The above results are for disabled random shuffling in training phase. The below table results are obtained using random shuffle.

sigma
of
f
\(\lambda \)learning
rate
number
of
epochs
batch
size
Q1Q2Q1
per
bubble
Q2
per
bubble
% gap
between training and
validation
exp110.011e-5300116.2417090.8175960.6381740.031777112
exp210.0051e-5300116.6629920.7748430.6570080.030192151
exp31.50.011e-5300116.1890290.8050560.6370.03141348.5
exp41.50.0051e-5300116.5655830.7722280.6526270.03014791.3
exp520.011e-5300117.0084820.8781610.6697870.03432922
exp620.0051e-5300118.7841880.9284130.7422850.03647554.6

Leave a Comment

Your email address will not be published. Required fields are marked *