PI: Jason Corso
Graduate Students: Albert Chen, Kevin Keane, David Johnson, Xin Li, Harsh Shah, Caiming Xiong
Undergraduate Students Alexander Haynie, Brian Borncamp

Funding: NSF IIS 0845282

From representation to learning to inference, effective use of high-level semantic knowledge in computer vision remains a challenge in bridging the signal-symbol gap. This research investigates the role of semantics in visual inference through the generalized image understanding problem: to automatically detect, localize, segment, and recognize the core high-level elements and how they interact in an image, and provide a parsimonious semantic description of the image.

Specifically, this research examines a unified methodology that integrates low- (e.g., pixels and features), mid- (e.g. latent structure), and high-level (e.g., semantics) elements for visual inference. Adaptive graph hierarchies induced directly from the images provide the core mathematical representation. A statistical interpretation of affinities between neighboring pixels and regions in the image drives this induction. Latent elements and structure are captured with multilevel Markov networks. A probabilistic ontology represents the core knowledge and uncertainty of the inferred structure and guides the ultimate semantic interpretation of the image. At each level, rigorous methods from computer science and statistics are connected to and combined with formal semantic methods from philosophy.

A symbiotic education plan involving graduate and undergraduate mentoring and education, professional tutorial courses at the boundary of vision and ontology, and K-12 outreach is incorporated into the research plan. The research and education, disseminated broadly through both the applied science and semantics/philosophy literatures, lays a foundation on which to both utilize and automatically extract rich semantic information from images and other signal data for critical application areas such as internet vision, autonomous navigation, and ambient biometrics.

A key enabler for reasoning about the relations between elements in a scene is a robust object detection mechanism. During the Summer of Code 2010, I have designed an experential learning summer workshop for a group of masters students and undergraduates in which we are implementing and evaluating diverse state of the art methods in object detection on large real-world image datasets. Information is available a the SOC2010 page. Results and code will be posted here.

• Quad-Chart describing the overall CAREER project.

[1]	S. Kumar, V. Dhiman, P. Koch, and J. J. Corso. Learning compositional sparse bimodal models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5):1032--1044, 2018. [ bib | DOI | code ]
[2]	C. Xiong, D. M. Johnson, and J. J. Corso. Active clustering with model-based uncertainty reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(1):5--17, 2017. Original Version: ArXiv 1402.1783. [ bib | .pdf ]
[3]	D. M. Johnson, C. Xiong, and J. J. Corso. Semi-supervised nonlinear distance metric learning via forests of max-margin cluster hierarchies. IEEE Transactions on Knowledge and Data Engineering, 28(4):1035--1046, 2016. [ bib | DOI | .pdf ]
[4]	C. Xu and J. J. Corso. Actor-action semantic segmentation with grouping-process models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016. [ bib | data ]
[5]	C. Xu and J. J. Corso. LIBSVX: A supervoxel library and benchmark for early video processing. International Journal of Computer Vision, 119:272--290, 2016. [ bib ]
[6]	R. Xu, C. Xiong, W. Chen, and J. J. Corso. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In Proceedings of AAAI Conference on Artificial Intelligence, 2015. [ bib | .pdf ]
[7]	J. Lu, R. Xu, and J. J. Corso. Human action segmentation with hierarchical supervoxel consistency. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2015. [ bib | .pdf ]
[8]	C. Xu, S.-H. Hsieh, C. Xiong, and J. J. Corso. Can humans fly? Action understanding with multiple classes of actors. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2015. [ bib | poster | data | .pdf ]
[9]	W. Chen and J. J. Corso. Action detection by implicit intentional motion clustering. In Proceedings of IEEE International Conference on Computer Vision, 2015. [ bib | poster | .pdf ]
[10]	W. Wu, A. Y. C. Chen, L. Zhao, and J. J. Corso. Brain tumor detection and segmentation in a CRF (conditional random fields) framework with pixel-pairwise affinity and superpixel-level features. International Journal of Computer Aided Radiology and Surgery, 9(2):241--253, 2014. [ bib | http ]
[11]	C. Xu, R. F. Doell, S. J. Hanson, C. Hanson, and J. J Corso. A study of actor and action semantic retention in video supervoxel segmentation. International Journal of Semantic Computing, 2014. Selected as a Best Paper from ICSC; an earlier version appeared as arXiv:1311.3318. [ bib | .pdf ]
[12]	C. Xiong, W. Chen, G. Chen, D. Johnson, and J. J. Corso. Adaptive quantization: An information-based approach to learning binary codes. In Proceedings of SIAM International Conference on Data Mining, 2014. [ bib | code | .pdf ]
[13]	V. Dhiman, A. Kundu, F. Dellaert, and J. J. Corso. Modern MAP inference methods for accurate and faster occupancy grid mapping on higher order factor graphs. In Proceedings of International Conference on Robotics and Automation, 2014. [ bib | code | .pdf ]
[14]	W. Chen, C. Xiong, R. Xu, and J. J. Corso. Actionness ranking with lattice conditional ordinal random fields. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2014. [ bib | poster | code | .pdf ]
[15]	S. Kumar, V. Dhiman, and J. J. Corso. Learning compositional sparse models of bimodal percepts. In Proceedings of AAAI Conference on Artificial Intelligence, 2014. [ bib | code | .pdf ]
[16]	C. Xiong, S. McCloskey, and J. J. Corso. Latent domains for visual domain adaptation. In Proceedings of AAAI Conference on Artificial Intelligence, 2014. [ bib ]
[17]	A. Barbu, D. Barrett, W. Chen, N. Siddharth, C. Xiong, J. J. Corso, C. D. Fellbaum, C. Hanson, S. J. Hanson, S. Hélie, E. Malaia, B. A. Pearlmutter, J. M. Siskind, T. M. Talavage, and R. B. Wilbur. Seeing is worse than believing: Reading people's minds better than computer-vision methods recognize actions. In Proceedings of European Conference on Computer Vision, 2014. [ bib | .pdf ]
[18]	J. J. Corso. Toward parts-based scene understanding with pixel-support parts-sparse pictorial structures. Pattern Recognition Letters: Special Issue on Scene Understanding and Behavior Analysis, 34(7):762--769, 2013. Early version appears as arXiv.org tech report 1108.4079v1. [ bib | .pdf ]
[19]	Y. Miao and J. J. Corso. Hamiltonian streamline guided feature extraction with application to face detection. Journal of Neurocomputing, 120:226--234, 2013. Early version appears as arXiv.org tech report 1108.3525v1. [ bib | http ]
[20]	J. A. Delmerico, D. Baran, P. David, J. Ryde, and J. J. Corso. Ascending stairway modeling from dense depth imagery for traversability analysis. In Proceedings of IEEE International Conference on Robotics and Automation, 2013. [ bib | project | .pdf ]
[21]	P. Das, C. Xu, R. F. Doell, and J. J. Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2013. [ bib | poster | data | .pdf ]
[22]	D. M. Johnson, C. Xiong, J. Gao, and J. J. Corso. Comprehensive cross-hierarchy cluster agreement evaluation. In Proceedings of AAAI Conference on Artificial Intelligence (Late-Breaking Papers Track), 2013. [ bib | code | .pdf ]
[23]	C. Xiong, D. M. Johnson, and J. J. Corso. Uncertainty reduction for active image clustering via a hybrid global-local uncertainty model. In Proceedings of AAAI Conference on Artificial Intelligence (Late-Breaking Papers Track), 2013. [ bib | .pdf ]
[24]	L. Zhao, W. Wu, and J. J. Corso. Semi-automatic brain tumor segmentation by constrained MRFs using structural trajectories. In Proceedings of Medical Image Computing and Computer Aided Intervention, 2013. [ bib | .pdf ]
[25]	V. Dhiman, J. Ryde, and J. J. Corso. Mutual localization: Two camera relative 6-dof pose estimation from reciprocal fiducial observation. In Proceedings of International Conference on Intelligent Robots and Systems, 2013. [ bib | slides | code | .pdf ]
[26]	C. Xu, R. F. Doell, S. J. Hanson, C. Hanson, and J. J Corso. Are actor and action semantics retained in video supervoxel segmentation? In Proceedings of IEEE International Conference on Semantic Computing, 2013. [ bib | .pdf ]
[27]	C. Xu, S. Whitt, and J. J. Corso. Flattening supervoxel hierarchies by the uniform entropy slice. In Proceedings of the IEEE International Conference on Computer Vision, 2013. [ bib | poster | project | video | .pdf ]
[28]	J. A. Delmerico, P. David, and J. J. Corso. Building facade detection, segmentation, and parameter estimation for mobile robot stereo vision. Image and Vision Computing, 31(11):841--852, 2013. [ bib | project | data | .pdf ]
[29]	C. Xu and J. J. Corso. Evaluation of super-voxel methods for early video processing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012. [ bib | code | project | .pdf ]
[30]	S. Sadanand and J. J. Corso. Action bank: A high-level representation of activity in video. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2012. [ bib | code | project | .pdf ]
[31]	G. Chen, C. Xiong, and J. J. Corso. Dictionary transfer for image denoising via domain adaptation. In Proceedings of IEEE International Conference on Image Processing, 2012. [ bib | .pdf ]
[32]	R. Xu, P. Agarwal, S. Kumar, V. N. Krovi, and J. J. Corso. Combining skeletal pose with local motion for human activity recognition. In Proceedings of VII Conference on Articulated Motion and Deformable Objects, 2012. [ bib | slides | .pdf ]
[33]	C. Xiong, D. Johnson, R. Xu, and J. J. Corso. Random forests for metric learning with implicit pairwise position dependence. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2012. [ bib | slides | code | .pdf ]
[34]	M. A. Bustamante and J. J. Corso. Using probabilistic ontologies for video exploration. In Proceedings of the Eighteenth Americas Conference on Information Systems, 2012. [ bib ]
[35]	C. Xiong and J. J. Corso. Coaction discovery: Segmentation of common actions across multiple videos. In Proceedings of Multimedia Data Mining Workshop in Conjunction with the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (MDMKDD), 2012. [ bib | .pdf ]
[36]	C. Xu, C. Xiong, and J. J. Corso. Streaming hierarchical video segmentation. In Proceedings of European Conference on Computer Vision, 2012. [ bib | code | project | .pdf ]
[37]	J. Ryde and J. J. Corso. Fast voxel maps with counting bloom filters. In Proceedings of International Conference on Intelligent Robots and Systems, 2012. [ bib | code | .pdf ]
[38]	J. A. Delmerico, J. J. Corso, D. Baran, P. David, and J. Ryde. Ascending stairway modeling: A first step toward autonomous multi-floor exploration. In Proceedings of IEEE/RSJ Intelligent Robots and Systems (Video Proceedings), 2012. [ bib | project | video ]
[39]	C. S. Lea and J. J. Corso. Efficient hierarchical markov random fields for object detection on a mobile robot. Technical Report 1111.1599v1, arXiv, November 2011. [ bib ]
[40]	Y. Miao and J. J. Corso. Hamiltonian streamline guided feature extraction with applications to face detection. Technical Report 1108.3525v1, arXiv, August 2011. [ bib ]
[41]	R. S. Alomari, J. J. Corso, and V. Chaudhary. Labeling of lumbar discs using both pixel- and object-level features with a two-level probabilistic model. IEEE Transactions on Medical Imaging, 30(1):1--10, 2011. [ bib | .pdf ]
[42]	A. Y. C. Chen and J. J. Corso. Temporally consistent multi-class video-object segmentation with the video graph-shifts algorithm. In Proceedings of the 2011 IEEE Workshop on Motion and Video Computing, 2011. [ bib | code | project | .pdf ]
[43]	D. Gagneja, C. Xiong, and J. J. Corso. Towards a parts-based approach to sub-cortical brain structure parsing. In Proceedings of SPIE Conference on Medical Imaging, 2011. [ bib | .pdf ]
[44]	D. R. Schlegel, A. Y. C. Chen, C. Xiong, J. A. Delmerico, and J. J.  Corso. AirTouch: Interacting with computer systems at a distance. In Proceedings of IEEE Winter Vision Meetings: Workshop on Applications of Computer Vision (WACV), 2011. [ bib | .pdf ]
[45]	J. A. Delmerico, P. David, and J. J. Corso. Building facade detection, segmentation, and parameter estimation for mobile robot localization and guidance. In Proceedings of International Conference on Intelligent Robots and Systems, 2011. [ bib | project | data | .pdf ]
[46]	A. Y. C. Chen and J. J. Corso. On the effects of normalization in adaptive MRF hierarchies. In Proceedings of CompImage '10---Computational Modeling of Objects Presented in Images, 2010. [ bib | .pdf ]
[47]	J. A. Delmerico, J. J. Corso, and P. David. Boosting with stereo features for building facade detection on mobile platforms. In Proceedings of Western New York Image Processing Workshop, 2010. [ bib | .pdf ]
[48]	A. Y. C. Chen and J. J. Corso. Propagating multi-class pixel labels throughout video frames. In Proceedings of Western New York Image Processing Workshop, 2010. [ bib | .pdf ]

This project is supported under NSF IIS 0845282: "CAREER: Generalized Image Understanding with Probabilistic Ontologies and Dynamic Adaptive Graph Hierarchies". Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.