Web Search and Data Mining

Course Description

Web Search and Data Mining (WiSDoM) is an area that aims at extracting knowledge from the largest source of information created by humans: The Web! Throughout this course we will see how this extracted knowledge can solve complex tasks with advanced Computer Vision, Natural Language and Information Retrieval algorithms. The main topics of this course are:

Text and visual data representation
Large Language Models
Billion scale text and image search
Visual question answering
Multimodal conversational assistants
Case study: ChatGPT

This course includes intensive hands-on laboratories where key CV, NLP and IR algorithms are examined.

Objectives

Learn what is an information embedding.
Learn the semantic associations between visual data, natural language data and user data.
Learn how to relate user information needs to actionable data.
Learn how to do a critical analysis of experimental results.
Develop autonomous and creative problem solving skills.

Grading

Exam (40%)
Project (45% = 15% phase 1 + 20% phase 2 + 10% phase 3)
Project originality (15%)

Schedule

08/mar Introduction
15/mar Web document categorization
22/mar Word embeddings
29/mar Transformer (encoder)
05/abr Billion scale indexing
12/abr Visual search
19/abr ExpoFCT / Farfetch talk
26/abr Vision-and-language models
03/mai Project support
10/mai Multimodal conversational assistants
17/mai Pre-trainning and fine-tunning
24/mai Transformer (decoder)
31/mai Project support
08/jun Project deadline / Test

Tutorials

Tutorial 0 - Environment setup instructions
Tutorial 1 - Text representation
Tutorial 2 - OpenSearch
Tutorial 3 - Transformer encoder -> UPDATED!
Tutorial 4 - Vision understanding
Tutorial 5 - Vision and language models
Tutorial 6a - Dialog state tracking (Amazon Alexa TaskBot)
Tutorial 6b - Dialog state tracking (Farfetch conversational assistant)
Tutorial 7 - Model fine tuning (draft) You will also need this extra code.
Tutorial 8 - Language generation (draft)

Lecturers

Joao Magalhaes ([email protected] - remove the ‘x’ character to send an email)