I. 3D Reconstruction in Capsule Endoscopy Videos using Shape from Shading (SfS) Techniques
Abstract:
Endoscopic optical systems have unique characteristics that differentiate from other optical imaging techniques. Especially, the near-light sources in capsule endoscopy cause the problem of inferring shape from shading (SfS) a complicated one. Moreover, specular and non-Lambertian nature of the intestinal tissues call for a non-traditional model. In this project, we study robust perspective SfS models and show how it helps reconstruct 3D shapes in an efficient manner for a variety of endoscopic systems.
Example reconstructions in wireless capsule endoscopy images:
The following examples illustrates the proposed shape from shading (SfS) based 3D reconstruction scheme [1] for different Wireless Capsule Endoscopy images:
Villus structures
Polyp
3D Reconstruction using our scheme [1]
Note that, the first example, a Wireless Capsule Endoscopy image, contains villus structures outlining the mucosa surface, the second example contains a polyp, and both are captured efficiently in our 3D reconstruction.
Reference:
[1]* V. B. S. Prasath, I. N. Figueiredo, P. N. Figueiredo, K. Palaniappan. Mucosal region detection and 3D reconstruction in wireless capsule endoscopic videos by active contours. 34th Annual International Conference IEEE EMBS (EMBC), San Diego, CA, 2012. Proc. IEEE, pp. 4014-4017.
Acknowledgement:
*Part of the work [1] was done while the first author was at the Department of Mathematics, University of Coimbra, PT. The first author also gratefully acknowledges Prof. Richard Tsai (UTAustin) for his advise and help.
*The work [1] was partially supported by the research project UTAustin/MAT/0009/2008 of the UT Austin|Portugal Program (http://www.utaustinportugal.org/) and by CMUC and FCT (Portugal), through European program COMPETE/FEDER.
II. Shape Reconstruction for Endoscopic Images with Robust Perspective Shape from Shading (RP-SfS)
Aim: Endoscopic optical systems have unique characteristics that differentiate from other optical imaging techniques. Especially, the near-light sources cause the problem of inferring shape from shading (SfS) a complicated one. Moreover, specular and non-Lambertian nature of the intestinal tissues call for a non-traditional model. In this work we propose a robust perspective SfS model and show how it reconstructs shapes in an efficient manner for a variety of endoscopic systems.
Example reconstruction in Colonoscopy image with a peduncled polyp:
In this example, a Colonoscopy image, has a big peduncle polyp and our 3D reconstruction shows the amount of its protrusion out of lumen. This is before an illumination correction step (images to be added) and hence the small bumps seen are due to specularities of the mucosa surface. A preliminary specular removal step will eliminate such bumps.
Example reconstruction in Colonoscopy image with a sessile polyp:
Note that, this example, a Colonoscopy image, contains a polyp and it is captured efficiently in our 3D reconstruction (after a specular removal step).
Comparison with related Perspective Shape from Shading schemes:
Sample result:
Input Colonoscopy image
Our result [1]
Ref [2]
References:
[1] V. B. S. Prasath, H. Kawanaka. Near-light perspective shape from shading for 3D visualizations in endoscopy systems. IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Kansas, USA. Proc. IEEE, pp. 2293-2295, Nov 2017. doi:10.1109/BIBM.2017.8218031. Download poster at figshare: 10.6084/m9.figshare.5526907
[2] V. B. S. Prasath. Mucosal surface reconstruction from endoscopic images with robust perspective shape from shading - From classical models to deep learning. In preparation, 2023.
Bibliography:
[2] M. Visentini-Scarzanella, D. Stoyanov, G.-Z. Yang, Metric Depth Recovery from Monocular Images Using Shape-from-Shading and Specularities, IEEE International Conference on Image Processing (ICIP), Orlando, FL, 2012.