Multimedia Retrieval

Multimedia retrieval consists of the theory, techniques, and tools that enable end users to search content of interest in large data collections consisting of (hyper)text, images, videos, audio, and 2D and 3D shapes. Multimedia retrieval is present in many daily applications, such as

searching the web for pages that contain a specific phrase
searching for images related to a given topic in the world wide web
finding images and 2D and 3D shapes similar to a given model
finding songs that match a given (hummed) melody or rhythm

Besides these, multimedia retrieval is also present in more specialized contexts such as

surveillance: finding frames in which the faces of given suspects is found
medical science: finding X-ray or CT scans that contain a given anatomic pattern
computer-aided design: finding parts that fit a given assembly

Of course, we cannot realistically cover all these domains and applications in one course. So, we will focus on a single case: How to retrieve 3D shapes from a shape collection. The principles, techniques, and largely software this course teaches are, however, the same for any multimedia retrieval task for other data types.

Course aims

The aim of this course is to provide an in-depth coverage of multimedia retrieval (MR) techniques. The course covers the following learning aims:

the MR operational pipeline: how MR works
data representation: how MR data is stored and manipulated
perception: how humans assimilate (perceive) MR data
feature extraction: how MR data is reduced to simple mathematical descriptors
matching: how to compare MR data using feature descriptors
evaluation: how to measure the quality of an MR system
scalability: how to deal with large MR databases
presentation: how to visually explore MR databases

Course audience

The targeted audience includes students and (young) researchers interested in designing, implementing, and optimizing MR systems. The students taking this course should have a background in

calculus, linear algebra, and statistics (differentiation, integration, matrix/vector operations, first-order differential equations, histograms/distributions)
computer science (general data structures and algorithms). Programming skills in a mainstream programming language (e.g. C, C++, Java, Python) are very important
computer graphics or shape modeling (meshes, normals, 3D transformations, basic 3D rendering).

Course details

See the sidebar for links to the course slides, assignment, grading, and additional material.