Intelligent Ocular Image Generation

How Our Project Will Contribute To The Growth Of The Decentralized AI Platform

This project works to develop appropriate libraries and processes for synthetic images. Next steps include models offered on the SingularityNET Platform as APIs/services with usage-based pricing models for healthcare providers and researchers and sliding scale for researchers facing financial challenges. 

Potential future revenue streams include:

1. Subscription fees from hospitals/clinics

2. Pay-per-use model for AI model inference

3. Custom model development/fine-tuning services

Our Team

We bring experienced, capable university faculty, development professionals, and managers with extensive background and capabilities in ML, AI, Statistics, and Scientific Computing together to develop algorithms and deliver a solid, robust, easy to use code base. All faculty have strong records of successful funded activity together with years of classroom experience at graduate and undergraduate levels, fluid communications skills, and administrative experience. This is a team set to deliver!

Company Name (if applicable)

Photrek https://www.photrek.io/home

The core problem we are aiming to solve

There is a critical lack of available data sets needed to train ML and AI solutions.

Applications are being developed to aid in the detection of diseases of the eye via fundus photographs and optical coherence tomography images [6]. Classifiers have been developed for conditions including detection of diabetic retinopathy [1], crossed eyes, bulging eyes, and conjunctivitis [2], macular oedema, age-related macular degeneration [3], loiasis and other parasites [4], cataracts, glaucoma, bulging eyes, and ocular hypertension [5]. Other approaches include eye disease classification using deep learning .

As these machine learning and artificial intelligence methods are developed to diagnose medical conditions, in particular those involving the human vision system, efforts are often hindered by the lack of available data sets on which to train algorithms [7] [8], [9]. 

We address this gap by developing methods for the synthetic generation of ocular images available to researchers in ML and AI and in particular to the SingularityNET community. 

Our specific solution to this problem

Photrek is keenly engaged in commercial grade data generation, with recent successes in image and video processing, and sees this as a natural area of application. 

This Deep Fund 4 New Projects Pool activity will 

• leverage Photrek’s existing competencies in synthetic data generation and commercial hosting to develop a model for web hosting of AI and ML applications; 

• work with the extensive professional networks of Photrek’s team to build a new research and development group across three continents, supporting talent development at four separate universities. We bring these diverse perspectives and expertise to bear on a problem of fundamental importance;

• engage the SingularityNET community, through public forums such as the DeepFund TownHall and other avenues for feedback in order to ensure that the project is meeting community needs; and 

• develop plans for future work and integrate this project with Photrek’s other SingularityNET efforts like Uncertain Knowledge Graphs by, for example, generating ocular images from text. This has proof of concept in other arenas such as Midjourney V5- text to image and  Da Vinci, though Foundational Models trained very broadly have difficulty producing the specific types of images needed.

Once a strong product is created and used fruitfully by the SNET community, we plan to work towards hosting on the SingularityNET AI Marketplace.

Project details

Our basic development processes will follow:

1. Data Preparation: Curate and preprocess the ocular image datasets, ensuring proper labeling and annotation of disease severity levels.

2. Training:

1. Variational Auto-Encoder (VAE) [12]: Train a VAE model on the real ocular image data. The VAE will learn to encode the images into a latent space representation and generate new images from this latent space. We extending the basic VAE to a conditional VAE by conditioning the generation process on the desired disease severity level. This allows us to generate images with specified severity levels. We bring Photrel Risk Aware methodologies by using and assessing the Coupled VAE in this context. 

2. Generative adversarial network (GAN) [7]: Implement the TrippleGAN architecture, which consists of three components: a generator, a discriminator, and a classifier. The generator will generate synthetic images, the discriminator will distinguish between real and generated images, and the classifier will classify the disease severity levels. Train the TrippleGAN in an adversarial manner, where the generator tries to produce realistic images that can fool the discriminator, while the discriminator tries to distinguish between real and generated images. The classifier will provide additional supervision by classifying the disease severity levels of both real and generated images.

3. Evaluation: Once the generator is trained, we generate a large dataset of synthetic ocular images with varying disease severity levels and evaluate the performance of disease classification algorithms on this synthetic dataset, comparing the results to their performance on real data.

4. Iterative Improvement: Use the evaluation results to identify weaknesses in the generated images or the classification algorithms. Refine the TrippleGAN architecture, training process, or classification models as needed, and repeat the evaluation process.

This approach leverages the power of VAEs for generating diverse and realistic ocular images, while the TrippleGAN architecture ensures that the generated images accurately represent the desired disease severity levels. The ability to create labeled synthetic data on-demand can be invaluable for training and testing ocular disease classification models, potentially improving their performance and robustness.

We anticipate following an ensemble strategy along the following lines. 

1. Data Preprocessing and Augmentation:

• Preprocess the ocular image data from various sources (e.g., fundus photographs, OCT scans, fluorescein angiography) to ensure consistency in image size, color space, and other relevant properties.

• Perform data augmentation techniques like rotation, flipping, scaling, and brightness adjustments to increase the diversity of the training data and improve the model's generalization capability.

2. Ensemble of Generative Models:

• Train multiple TripleGAN models on different subsets of the ocular image data, each focusing on specific types of ocular images or disease conditions.

• Employ different architectures, hyperparameters, and initialization strategies for each TripleGAN model in the ensemble to capture diverse representations and features.

3. Ensemble of Discriminators and Classifiers:

• For each TripleGAN model in the ensemble, train multiple discriminators and classifiers with different architectures, loss functions, and regularization techniques.

• This diversity in the discriminators and classifiers can help capture various aspects of the image data, leading to more robust and accurate evaluation of the generated images.

4. Ensemble Strategy:

• During inference, generate images from each TripleGAN model in the ensemble, and combine the outputs using an appropriate ensemble strategy.

• Possible ensemble strategies include averaging, majority voting, stacking, or more advanced techniques like mixture of experts or boosting.

5. Ensemble Calibration and Refinement:

• Evaluate the performance of the ensemble on a held-out validation set containing diverse ocular image types and disease conditions.

• Use the evaluation results to calibrate the ensemble weights, retrain or fine-tune individual models, or adjust the ensemble strategy as needed.

• Iteratively refine the ensemble until satisfactory performance is achieved across all types of ocular images.

6. Continuous Learning and Adaptation:

• Implement a continuous learning approach to update and adapt the ensemble as new ocular image data becomes available.

• Periodically retrain or fine-tune individual models in the ensemble or introduce new models to maintain the ensemble's relevance and performance over time.

By employing an ensemble of TripleGAN models, discriminators, and classifiers, we leverage the strengths of different architectures and capture diverse representations of the ocular image data. This approach improves the generalization capability of the overall system, enabling it to work effectively on various types of ocular images while maintaining novelty and diversity in the generated samples.

Data Acquisition and Ethics

Obtaining ocular image data, especially involving patient information, requires careful handling from an ethical standpoint.  Researchers typically must partner with hospitals or clinics and obtain approval from ethics review boards to access de-identified/anonymized patient data for research purposes. 

For this phase of the project, we will use publicly available datasets like those from Kaggle competitions or research repositories, which have already gone through ethical vetting.

Exploring synthetic data generation approaches as in this proposal minimizes reliance on real patient data during development/testing. Regardless of the approach, maintaining data privacy, obtaining informed consent where applicable, and following ethical guidelines established by organizations like the World Medical Association will be crucial.

Technical Feasibility

For image processing, state-of-the-art deep learning models like convolutional neural networks (CNNs) and vision transformers will be employed. Specific architectures like EfficientNets, ResNets, etc. will be customized for the ocular disease classification task.

For generative modeling, techniques like Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and more recent approaches like Diffusion Models could be leveraged. These are established methods and we do not anticipate technical challenges beyond the implementation process.  

Cloud computing resources (TPUs/GPUs) will provide the computational power required to train these large AI models efficiently.

User Interaction Design

The user interface should prioritize ease of use for medical professionals. We will work towards a  simple upload interface for ocular images, with clear result visualization.

Advanced users could be provided with options to adjust parameters like desired disease severity levels. Interactive visualizations could allow users to explore the model's decision-making process.

Following accessibility guidelines and involving target users through user testing would be critical for an effective design.

Bibliography

[1] V. Gulshan et al., “Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs,” JAMA, vol. 316, no. 22, p. 2402, Dec. 2016, doi: 10.1001/jama.2016.17216.

[2] S. K. Sattigeri, “EYE DISEASE IDENTIFICATION USING DEEP LEARNING,” vol. 09, no. 07, 2022.

[3] D. S. W. Ting et al., “Artificial intelligence and deep learning in ophthalmology,” Br. J. Ophthalmol., vol. 103, no. 2, pp. 167–175, Feb. 2019, doi: 10.1136/bjophthalmol-2018-313173.

[4] S. Kumar, T. Arif, A. S. Alotaibi, M. B. Malik, and J. Manhas, “Advances Towards Automatic Detection and Classification of Parasites Microscopic Images Using Deep Convolutional Neural Network: Methods, Models and Research Directions,” Arch. Comput. Methods Eng., vol. 30, no. 3, pp. 2013–2039, Apr. 2023, doi: 10.1007/s11831-022-09858-w.

[5] A. A. Marouf, M. M. Mottalib, R. Alhajj, J. Rokne, and O. Jafarullah, “An Efficient Approach to Predict Eye Diseases from Symptoms Using Machine Learning and Ranker-Based Feature Selection Methods,” Bioengineering, vol. 10, no. 1, p. 25, Dec. 2022, doi: 10.3390/bioengineering10010025.

[6] T. Babaqi, M. Jaradat, A. E. Yildirim, S. H. Al-Nimer, and D. Won, “Eye Disease Classification Using Deep Learning Techniques.” arXiv, Jul. 19, 2023. Accessed: Mar. 27, 2024. [Online]. Available: http://arxiv.org/abs/2307.10501

[7] M. Fan, X. Peng, and X. Gong, “Ocular Disease Recognition and Classification using TripleGAN,” in Proceedings of the 2023 8th International Conference on Biomedical Signal and Image Processing, Chengdu China: ACM, Jul. 2023, pp. 7–11. doi: 10.1145/3613307.3613309.

[8] “Jin et al. - 2022 - FIVES A Fundus Image Dataset for Artificial Intel.pdf.”

[9] Z. Yu, Q. Xiang, J. Meng, C. Kou, Q. Ren, and Y. Lu, “Retinal image synthesis from multiple-landmarks input with generative adversarial networks,” Biomed. Eng. OnLine, vol. 18, no. 1, p. 62, Dec. 2019, doi: 10.1186/s12938-019-0682-x.

[10] “Shang et al. - 2024 - SynFundus-1M A High-quality Million-scale Synthet.pdf.”

[11] R. Nuzzi, G. Boscia, P. Marolo, and F. Ricardi, “The Impact of Artificial Intelligence and Deep Learning in Eye Diseases: A Review,” Front. Med., vol. 8, p. 710329, Aug. 2021, doi: 10.3389/fmed.2021.710329.

[12] D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Found. Trends® Mach. Learn., vol. 12, no. 4, pp. 307–392, 2019, doi: 10.1561/2200000056.

Competition and USPs

The inspiration for this work stems partially from the participation of Photrek researchers Thistleton and Nelson in the National Science Foundation's Innovation Corps (I-Corps™). Compelling evidence gained from extensive conversations with leaders in industry and academia during the Customer Discovery process  impressed upon the Photrek team the lack of available data sets for development of new Machine Learning and Artificial Intelligence techniques across a variety of fields. 

It is difficult to find any commercial firms providing access to the “on demand” or "bespoke" synthetic images we propose to allow users to generate, so we believe Photrek and SingularityNET will be well positioned with this product.

Existing resources

We build this work on code and expertise developed for the DF3 Simulating a Risky World and code and expertise developed in Photrek’s Coupled-Variational AutoEncoder project. 

Links and references

Please see Long Description for references in the scientific literature. Followiing are data sets and online resources.

New England Research Cloud

 Loiasis dataset of africa

Eye Dataset

Ocular Eye disease

Diabetic Retinopathy

Loiasis

Eye Anemia Dataset

Fundus images

Bajwa Hospital (Multi Eye disease dataset)

Multi-Label Retinal Diseases (MuReD) dataset

Structured Analysis of Retina Dataset

Digital Retinal Images for Vessel Extraction

Eye Datasets