Web Search and Data Mining

Course Description

Web Search and Data Mining (WiSDoM) is an area that aims at extracting knowledge from the largest source of information created by humans: The Web! Throughout this course we will see how this extracted knowledge can solve complex tasks with advanced Computer Vision, Natural Language and Information Retrieval algorithms. The main topics of this course are:

Text and visual data representation
Billion scale text and image search
Visual question answering
Multimodal conversational assistants
Recommender systems

This course includes intensive hands-on laboratories where key CV, NLP and IR algorithms are examined.

Objectives

Learn what is an information embedding.
Learn the semantic associations between visual data, natural language data and user data.
Learn how to relate user information needs to actionable data.
Learn how to do a critical analysis of experimental results.
Develop autonomous and creative problem solving skills.

Grading

Exam (40%)
Project (45% = 15% phase 1 + 20% phase 2 + 10% phase 3)
Project originality (15%)

Schedule

09/mar/21 Introduction
16/mar/22 Web document categorization
23/mar/22 Word embeddings
30/mar/22 Transformer (encoder)
06/abr/22 Billion scale indexing
13/abr/22 Visual search
20/abr/22 Vision-and-language models
27/abr/22 ExpoFCT
04/mai/22 Multimodal conversational assistants
11/mai/22 Pre-trainning and fine-tunning
18/mai/22 Transformer (decoder)
25/mai/22 Recommender systems
01/jun/22 HuggingFace invited lecture
08/jun/22 Project discussion / Test

Tutorials

We suggest you to use the account in the lab cluster. However, if you would like to have your own setup, you can follow this guide:

Tutorial 0 - Environment setup instructions
Tutorial 1 - Text representation
Tutorial 2 - OpenSearch
Tutorial 3 - Transformer encoder
Tutorial 4 - Vision understanding
Tutorial 5 - Vision and language models
Tutorial 6 - Dialog intent detection
Tutorial 7 - Model fine tuning (draft) You will also need this extra code.
Tutorial 8 - Language generation (draft)

Lecturers

Joao Magalhaes ([email protected] - remove the ‘x’ character to send an email)