Abstract

Medical imaging is a crucial tool in clinics to monitor tumor treatment progress. In practice, many imaging tools (such as magnetic resonance imaging (MRI) and computed tomography (CT) scans) are in general costly and may also expose patients to radiation, leading to potential side effects. Recent studies have demonstrated that ultrasound imaging, which is safe, low-cost, and easy to access, can monitor the drug delivery progress in solid tumors. However, the noisy nature of ultrasound images and the high-level uncertainty of cancer disease progression are still challenging in ultrasound-based tumor treatment monitoring. To overcome these barriers, this work presents a comparative study to explore the potential advantages of the emerging diffusion generative models against the commonly applied state-of-the-art generative models. Namely, the denoising diffusion models (DDMs), against the generative adversarial networks (GAN), and variational auto-encoders (VAE), are used for analyzing the ultrasound images through image augmentation. These models are evaluated based on their capacity to augment ultrasound images for exploring the potential variations of tumor treatment monitoring. The results across different cases indicate that the denoising diffusion implicit models (DDIM)/kernel inception distance (KID)-inception score (IS) model leveraged in this work outperforms the other models in the study in terms of similarity, diversity, and predictive accuracy. Therefore, further investigation of such diffusion generative models could be considered as they can potentially serve as a great predictive tool for ultrasound image-enabled tumor treatment monitoring in the future.

References

1.
NIH News in Health
,
2019
, “
Medical Scans Explained
,” NIH News in Health, Bethesda, MD, accessed Dec. 7, 2023, https://newsinhealth.nih.gov/2019/11/medical-scans-explained
2.
Priester
,
M. I.
, and
ten Hagen
,
T. L. M.
,
2023
, “
Image-Guided Drug Delivery in Nanosystem-Based Cancer Therapies
,”
Adv. Drug Delivery Rev.
,
192
, p.
114621
.10.1016/j.addr.2022.114621
3.
Mirniaharikandehei
,
S.
,
VanOsdol
,
J.
,
Heidari
,
M.
,
Danala
,
G.
,
Sethuraman
,
S. N.
,
Ranjan
,
A.
, and
Zheng
,
B.
,
2019
, “
Developing a Quantitative Ultrasound Image Feature Analysis Scheme to Assess Tumor Treatment Efficacy Using a Mouse Model
,”
Sci. Rep.
,
9
(
1
), p.
7293
.10.1038/s41598-019-43847-7
4.
Ektate
,
K.
,
Kapoor
,
A.
,
Maples
,
D.
,
Tuysuzoglu
,
A.
,
VanOsdol
,
J.
,
Ramasami
,
S.
, and
Ranjan
,
A.
,
2016
, “
Motion Compensated Ultrasound Imaging Allows Thermometry and Image Guided Drug Delivery Monitoring From Echogenic Liposomes
,”
Theranostics
,
6
(
11
), pp.
1963
1974
.10.7150/thno.15922
5.
Bharti
,
P.
, and
Mittal
,
D.
,
2020
, “
An Ultrasound Image Enhancement Method Using Neutrosophic Similarity Score
,”
Ultrason Imaging
,
42
(
6
), pp.
271
283
.10.1177/0161734620961005
6.
Liu
,
C.
,
Kapoor
,
A.
,
VanOsdol
,
J.
,
Ektate
,
K.
,
Kong
,
Z.
, and
Ranjan
,
A.
,
2018
, “
A Spectral Fiedler Field-Based Contrast Platform for Imaging of Nanoparticles in Colon Tumor
,”
Sci. Rep.
,
8
(
1
), p.
11390
.10.1038/s41598-018-29675-1
7.
Li
,
Y.
,
VanOsdol
,
J.
,
Ranjan
,
A.
, and
Liu
,
C.
,
2022
, “
A Multilayer Network-Enabled Ultrasonic Image Series Analysis Approach for Online Cancer Drug Delivery Monitoring
,”
Comput. Methods Programs Biomed.
,
213
, p.
106505
.10.1016/j.cmpb.2021.106505
8.
Gao
,
Y.
,
Maraci
,
M. A.
, and
Noble
,
J. A.
,
2016
, “
Describing Ultrasound Video Content Using Deep Convolutional Neural Networks
,”
IEEE 13th International Symposium on Biomedical Imaging
(
ISBI
), Prague, Czech Republic, Apr. 13–16, pp.
787
790
.10.1109/ISBI.2016.7493384
9.
Atrey
,
K.
,
Singh
,
B. K.
, and
Bodhey
,
N. K.
,
2023
, “
Multimodal Classification of Breast Cancer Using Feature Level Fusion of Mammogram and Ultrasound Images in Machine Learning Paradigm
,”
Multimed. Tools Appl.
,
83
(
7
), pp.
21347
21368
.10.1007/s11042-023-16414-6
10.
Shijie
,
J.
,
Ping
,
W.
,
Peiyi
,
J.
, and
Siping
,
H.
,
2017
, “
Research on Data Augmentation for Image Classification Based on Convolution Neural Networks
,” Chinese Automation Congress (
CAC
), Jinan, China, Oct. 20–22, pp.
4165
4170
.10.1109/CAC.2017.8243510
11.
Chawla
,
N. V.
,
Bowyer
,
K. W.
,
Hall
,
L. O.
, and
Kegelmeyer
,
W. P.
,
2002
, “
SMOTE: Synthetic Minority Over-Sampling Technique
,”
J. Artif. Intell. Res.
,
16
, pp.
321
357
.10.1613/jair.953
12.
Cao
,
H.
,
Tan
,
C.
,
Gao
,
Z.
,
Chen
,
G.
,
Heng
,
P.-A.
, and
Li
,
S. Z.
,
2024
, “
A Survey on Generative Diffusion Model
,”
IEEE Transac. Knowled. Data Eng.
,
36
(
7
), pp.
2814
2830
.10.1109/TKDE.2024.3361474
13.
Devi
,
M. K.
,
Alias
,
A.
, and
Suganthi
,
K.
,
2021
, “
Review of Medical Image Synthesis Using GAN Techniques
,”
ITM Web Conf.
,
37
, p.
01005
.10.1051/itmconf/20213701005
14.
Kazerouni
,
A.
,
Aghdam
,
E. K.
,
Heidari
,
M.
,
Azad
,
R.
,
Fayyaz
,
M.
,
Hacihaliloglu
,
I.
, and
Merhof
,
D.
,
2023
, “
Diffusion Models in Medical Imaging: A Comprehensive Survey
,”
Medical Image Anal.
,
88
, p.
102846
.10.1016/j.media.2023.102846
15.
Qin
,
X.
,
Bui
,
F. M.
,
Nguyen
,
H. H.
, and
Han
,
Z.
,
2022
, “
Learning From Limited and Imbalanced Medical Images With Finer Synthetic Images From GANs
,”
IEEE Access
,
10
, pp.
91663
91677
.10.1109/ACCESS.2022.3202560
16.
Rapoport
,
N.
,
Kennedy
,
A. M.
,
Shea
,
J. E.
,
Scaife
,
C. L.
, and
Nam
,
K.-H.
,
2010
, “
Ultrasonic Nanotherapy of Pancreatic Cancer: Lessons From Ultrasound Imaging
,”
Mol. Pharmaceutics
,
7
(
1
), pp.
22
31
.https://pubs.acs.org/doi/full/10.1021/mp900128x
17.
Bau
,
D.
,
Zhu
,
J.-Y.
,
Wulff
,
J.
,
Peebles
,
W.
,
Strobelt
,
H.
,
Zhou
,
B.
, and
Torralba
,
A.
,
2019
, “
Seeing What a GAN Cannot Generate
,”
Proceedings of the IEEE/CVF International Conference on Computer Vision
, Seoul, Korea, Oct. 27–Nov. 2, pp.
4502
4511
.https://openaccess.thecvf.com/content_ICCV_2019/html/Bau_Seeing_What_a_GAN_Cannot_Generate_ICCV_2019_paper.html
18.
Hajij
,
M.
,
Zamzmi
,
G.
,
Paul
,
R.
, and
Thukar
,
L.
,
2022
, “
Normalizing Flow for Synthetic Medical Images Generation
,” IEEE Healthcare Innovations and Point of Care Technologies (
HI-POCT
), Houston, TX, Mar. 10–11, pp.
46
49
.10.1109/HI-POCT54491.2022.9744072
19.
Teo
,
C.
,
Abdollahzadeh
,
M.
,
Cheung
., and
N.-M.
,
(Man)
,
2023
, “
On Measuring Fairness in Generative Models
,”
Adv. Neural Inf. Process. Syst.
,
36
, pp.
10644
10656
.https://proceedings.neurips.cc/paper_files/paper/2023/hash/220165f9c7f51163b73c8c7fff578b4e-Abstract-Conference.html
20.
Yangue
,
E.
,
Fullington
,
D.
,
Smith
,
O.
,
Tian
,
W.
, and
Liu
,
C.
,
2024
, “
Diffusion Generative Model-Based Learning for Smart Layer-Wise Monitoring of Additive Manufacturing
,”
ASME J. Comput. Inf. Sci. Eng.
,
24
(
6
), p.
060903
.10.1115/1.4065092
21.
Ho
,
J.
,
Jain
,
A.
, and
Abbeel
,
P.
,
2020
, “
Denoising Diffusion Probabilistic Models
,”
Advances in Neural Information Processing Systems
,
Curran Associates, Inc.
, Red Hook, NY, pp.
6840
6851
.
22.
Song
,
J.
,
Meng
,
C.
, and
Ermon
,
S.
,
2022
, “
Denoising Diffusion Implicit Models
,”
arXiv:2010.02502v4
.10.48550/arXiv.2010.02502
23.
Akbar
,
M. U.
,
Wang
,
W.
, and
Eklund
,
A.
,
2023
, “
Beware of Diffusion Models for Synthesizing Medical Images—A Comparison With Gans in Terms of Memorizing Brain MRI and Chest X-Ray Images
,”
arXiv:2305.07644v3
.10.48550/arXiv.2305.07644
24.
Salimans
,
T.
,
Goodfellow
,
I.
,
Zaremba
,
W.
,
Cheung
,
V.
,
Radford
,
A.
,
Chen
,
X.
, and
Chen
,
X.
,
2016
, “
Improved Techniques for Training GANs
,”
Advances in Neural Information Processing Systems
,
Curran Associates, Inc.
, Red Hook, NY.
25.
Bińkowski
,
M.
,
Sutherland
,
D. J.
,
Arbel
,
M.
, and
Gretton
,
A.
,
2021
, “
Demystifying MMD GANs
,”
arXiv:1801.01401v5
.10.48550/arXiv.1801.01401
26.
Goodfellow
,
I.
,
Pouget-Abadie
,
J.
,
Mirza
,
M.
,
Xu
,
B.
,
Warde-Farley
,
D.
,
Ozair
,
S.
,
Courville
,
A.
, and
Bengio
,
Y.
,
2020
, “
Generative Adversarial Networks
,”
Commun. ACM
,
63
(
11
), pp.
139
144
.10.1145/3422622
27.
Arora
,
S.
,
Ge
,
R.
,
Liang
,
Y.
, and
Ma
,
T.
,
2017
, “
Generalization and Equilibrium in Generative Adversarial Nets (GANs)
,”
Proceedings of the 34th International Conference on Machine Learning, PMLR
, Sydney, Australia, Aug. 6–11, pp.
224
232
.
28.
Nagarajan
,
V.
,
Raffel
,
C.
, and
Goodfellow
,
I. J.
, “
Theoretical Insights Into Memorization in GANs
,” Neural Information Processing Systems Workshop, Vol. 1, p.
3
.https://vaishnavh.github.io/papers/GAN_memorization.pdf
29.
Arjovsky
,
M.
,
Chintala
,
S.
, and
Bottou
,
L.
,
2017
, “
Wasserstein Generative Adversarial Networks
,”
Proceedings of the 34th International Conference on Machine Learning, PMLR
, Sydney, Australia, Aug. 6–11, pp.
214
223
.https://proceedings.mlr.press/v70/arjovsky17a.html
30.
Radford
,
A.
,
Metz
,
L.
, and
Chintala
,
S.
,
2016
, “
Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks
,”
arXiv:1511.06434v2
.10.48550/arXiv.1511.06434
31.
Mao
,
X.
,
Li
,
Q.
,
Xie
,
H.
,
Lau
,
R. Y. K.
,
Wang
,
Z.
, and
Smolley
,
S. P.
,
2017
, “
Least Squares Generative Adversarial Networks
,”
Proceedings of the IEEE International Conference on Computer Vision
, pp.
2794
2802
.10.1109/ICCV.2017.304
32.
Karras
,
T.
,
Aittala
,
M.
,
Hellsten
,
J.
,
Laine
,
S.
,
Lehtinen
,
J.
, and
Aila
,
T.
,
2020
, “
Training Generative Adversarial Networks With Limited Data
,”
Advances in Neural Information Processing Systems
,
H.
Larochelle
,
M.
Ranzato
,
R.
Hadsell
,
M. F.
Balcan
, and
H.
Lin
, eds.,
Curran Associates, Inc
., Red Hook, NY, pp.
12104
12114
.
33.
Doersch
,
C.
,
2021
, “
Tutorial on Variational Autoencoders
,”
arXiv:1606.05908v3
.10.48550/arXiv.1606.05908
34.
van den Oord
,
A.
,
Vinyals
,
O.
, and
Kavukcuoglu
,
K.
,
2017
, “
Neural Discrete Representation Learning
,”
Advances in Neural Information Processing Systems
,
Curran Associates, Inc
., Red Hook, NY.https://proceedings.neurips.cc/paper_files/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf
35.
Mi
,
L.
,
Shen
,
M.
, and
Zhang
,
J.
,
2018
, “
A Probe Towards Understanding GAN and VAE Models
,”
arXiv:1812.05676v2
.10.48550/arXiv.1812.05676
36.
Zhang
,
R.
,
Isola
,
P.
,
Efros
,
A. A.
,
Shechtman
,
E.
, and
Wang
,
O.
,
2018
, “
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
,”
31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
, CVPR, Salt Lake City, June 18–22, pp.
586
595
.10.1109/CVPR.2018.00068
37.
Sheikh
,
H. R.
, and
Bovik
,
A. C.
,
2006
, “
Image Information and Visual Quality
,”
IEEE Trans. Image Process.
,
15
(
2
), pp.
430
444
.10.1109/TIP.2005.859378
38.
Wang
,
Z.
,
Simoncelli
,
E. P.
, and
Bovik
,
A. C.
,
2003
, “
Multiscale Structural Similarity for Image Quality Assessment
,”
The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers
, Vol.
2
, Pacific Grove, CA, Nov. 9–12, pp.
1398
1402
.10.1109/ACSSC.2003.1292216
39.
Thum
,
C.
,
1984
, “
Measurement of the Entropy of an Image With Application to Image Focusing
,”
Optica Acta: Int. J. Opt.
,
31
(
2
), pp.
203
211
.10.1080/713821475
40.
Shahapure
,
K. R.
, and
Nicholas
,
C.
,
2020
, “
Cluster Quality Analysis Using Silhouette Score
,” IEEE 7th International Conference on Data Science and Advanced Analytics (
DSAA
), Sydney, Australia, Oct. 6–9, pp.
747
748
.10.1109/DSAA49011.2020.00096
41.
VanOsdol
,
J.
,
Ektate
,
K.
,
Ramasamy
,
S.
,
Maples
,
D.
,
Collins
,
W.
,
Malayer
,
J.
, and
Ranjan
,
A.
,
2017
, “
Sequential HIFU Heating and Nanobubble Encapsulation Provide Efficient Drug Penetration From Stealth and Temperature Sensitive Liposomes in Colon Cancer
,”
J. Controlled Release
,
247
, pp.
55
63
.10.1016/j.jconrel.2016.12.033
You do not currently have access to this content.