PhD - Simultaneous estimation of object shape, location and speed using a light-field camera

JOB TITLE

PhD – Simultaneous estimation of object shape, location and speed using a light-field camera

EMPLOYER

UBFC – 32 avenue de l’Observatoire – 25000 BESANCON

LOCATION

VIBOT EMR CNRS 6000, ImViA
Iut Le Creusot,
12 rue de la fonderie
71200 Le Creusot Institut FEMTO-ST, CNRS UMR6174 / UFC / ENSMM / UTBM
Département Automatique et Systèmes Micro-Mécatroniques (AS2M)
24, rue Alain Savary
Besançon 25000 France

CONTRACT DURATION

36 months

DEADLINE APPLICATION

May 22, 2021

STARTING JOB

October 1, 2021

JOB DESCRIPTION

Light-field cameras, consisting of a matrix of microlenses placed in front of the photosensitive sensor, make it possible to capture the light intensity and orientation of the rays in a single shot. Thus, with a reduced space, it is possible to reconstruct the 3D structure of the observed scene and/or to focus the image at a distance chosen by the user [2]. In particular, within the framework of research in intracorporeal micro robotics [3], this type of camera could allow to reduce the size of endoscopes, so as to leave more space for surgical tools, without degrading the visual feedback provided to the surgeon.
The miniaturization of the sensor, not addressed in this thesis, requires the choice of a rolling shutter sensor. This kind of sensor suffers from optical distortions when the observed object moves in front of the sensor (see examples in [4]) but also allows to simultaneously localize the object and to estimate its speed, if the shape is known [5]. Nevertheless, when the object is of unknown shape, there remains an ambiguity about the separation of shape and motion [6] that a light-field camera should be able to resolve.Thus, in this PhD work, the aim is to take advantage of the particularity of these cameras which acquire the light intensity and the direction of the rays in a sequential way. The association of a sequential acquisition sensor and a microlens array has not been explored so far from an algorithmic point of view. To do so, two steps are essential to complete this project: the image processing of Light-field images and the design of specific geometrical computer vision tools.

First of all, it will be necessary to take into account the physics of the sensors to develop adapted low-level image processing. At each shot, this sensor provides us with a series of images with different focal lengths. Previously, we developed techniques to deal with this block of images [10,11] or to analyze the image of the associated epipolar plane [12] by using classical filtering techniques. In this work, we propose to take into account the specificity of acquisition of these images to develop new robust filtering methods adapted to the sensor. These adapted filtering methods will also allow us to rethink convolutional neural networks by taking inspiration from the work we have done on spherical [7], RGBD [8] and multimodal [9] sensors.
Then, the work will consist in proposing methods of geometrical 3D vision taking into account the microlens grid [13] and the sequential acquisition [6] in order to simultaneously estimate the shape of the unknown object filmed, its speed and its position.

This work can be applied in intracorporeal microbotics.
This work is supported by the iFilter project, funded by the French “Investissements d’Avenir” program, under the ISITE-BFC project (ANR-15-IDEX-0003).References
[1] E. H. Adelson and J. Y. A. Wang. Single Lens Stereo with Plenoptic Camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No. 2, pp. 99-106, February 1992.
[2] C. Hahne, A. Aggoun, and V. Velisavljevic, S. Fiebig, and M. Pesch “Baseline and triangulation geometry in a standard plenoptic camera,” Int. J. of Comput. Vis. (IJCV), 2017.
[3] N. Andreff et al. Micronanorobotique biomédicale. Grand Prix Scientifique de la Fondation Charles Defforey, Institut de France, 2018
[4] L. Magerand. Calcul de pose dynamique avec les caméras CMOS utilisant une acquisition séquentielle. Thèse de Doctorat. Université Blaise Pascal. 2014
[5] Omar Ait Aider, Nicolas Andreff, Jean-Marc Lavest, Philippe Martinet. Simultaneous Object Pose and Velocity Computation Using a Single View from a Rolling Shutter Camera. 9th European Conference on Computer Vision, ECCV’06, 2006, Graz, Austria.
[6] Lao, Yizhen & Ait-Aider, Omar & Bartoli, Adrien. Solving Rolling Shutter 3D Vision Problems using Analogies with Non-rigidity. International Journal of Computer Vision. 2021.
[7] Fernandez-Labrador, C., Facil, J. M., Perez-Yus, A., Demonceaux, C., Civera, J., & Guerrero, J. J. Corners for layout: End-to-end layout recovery from 360 images. IEEE Robotics and Automation Letters, 5(2), 1255-1262, 2020
[8] Wu, Z., Allibert, G., Stolz, C., & Demonceaux, C. Depth-Adapted CNN for RGB-D cameras. In Proceedings of the Asian Conference on Computer Vision, 2020 (Oral Presentation).
[9] Piasco, N., Sidibé, D., Gouet-Brunet, V., & Demonceaux, C. Improving image description with auxiliary modality for visual localization in challenging conditions. International Journal of Computer Vision, 129(1), 185-202, 2021.
[10] Yoon, Y., Jeon, H. G., Yoo, D., Lee, J. Y., & So Kweon, I. Learning a deep convolutional network for light-field image super-resolution. In Proceedings of the IEEE international conference on computer vision workshops, 2015.
[11] Farrugia, R. A., & Guillemot, C. Light field super-resolution using a low-rank prior and deep convolutional neural network. IEEE transactions on pattern analysis and machine intelligence, 42(5), 1162-1175, 2019
[12] Wu, G., Zhao, M., Wang, L., Dai, Q., Chai, T., & Liu, Y. Light field reconstruction using deep convolutional network on EPI. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[13] Labussière, M., Teulière, C., Bernardin, F., & Ait-Aider, O. Blur Aware Calibration of Multi-Focus Plenoptic Camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020

SUPERVISORS

Demonceaux Cédric – cedric.demonceaux@ubfc.fr
Andreff Nicolas – nicolas.andreff@ubfc.fr

QUALIFICATIONS

Master

CANDIDATE PROFILE

M.Sc. (or equivalent) in applied mathematics, computer science or robotics.
Solid background in computer vision.
Good skills in mathematical formalization, programming and experimentation.
Good dialogue skills.
Creativity and methodological rigor.
Oral and written communication skills in English and French.

APPLICATION

Please send the following documents (all in one PDF file) by e-mail to cedric.demonceaux@ubfc.fr and nicolas.andreff@ubfc.fr:

1) For EU candidates: Copy of your national ID card or of your passport page where your photo is printed.
For non-EU candidates: Copy of your passport page where your photo is printed.2) Curriculum Vitae (may include hyperlinks to your ResearchID, Research Gate Google Scholar accounts).

3) Detailed list of publications (may include hyperlinks to DOI of publications).

4) Letter of motivation relatively to the position (Cover Letter) in which applicants describe themselves and their contributions to previous research projects (maximum 2 pages)

5) Copy of your PhD degree if already available.

6) Coordinates of reference persons (maximum 3, at least your master thesis supervisor):
Title, Name, organization, e-mail.

If you have questions regarding the application, please contact the supervisors.