Information Theory
Code
12104
Academic unit
Faculdade de Ciências e Tecnologia
Department
Departamento de Informática
Credits
6.0
Teacher in charge
João Carlos Gomes Moura Pires, Pedro Manuel Corrêa Calvente Barahona
Weekly hours
4
Total hours
60
Teaching language
Português
Objectives
Knowledge
- The main Information Theory concepts, including Entropy, Information, Condicional Entropy, Channel Capacity.
- Identify these concepts in different contexts of communication systems, Storage, Data Processing and Inference.
- The main Information Theory''''s Theorem, the source coding theorem (with and without noise), its role, impact and application areas.
- The core aspects of Compression and error correcting codes.
- General principles and approaches to cryptography.
- Information Theory''''s application examples to different knowledge areas.
Application
- Apply Entropy and Information concepts to Computer Science and Machine Learning.
- Develop the main components of data compression algorithms or error correcting codes.
Soft-Skills
- Improve you ability to read and understand papers with a significant formal component. Be able to provide examples that illustrate the concepts and techniques discussed.
- Improve your team-work skills
- Improve your communication skills (oral and written) on formal subjects.
- Propose and develop simple but formal notation.
Prerequisites
Basic knowledge of Probability and Statistics.
Basic knowledge and Practice of computer programming.
Subject matter
1 - Introdution
- Main problems addressed by Information Theory and its creation.
- Relations with other bodies of knowledge.
- Information Theory Overview.
2 - Foundational Concepts
- Entropy for discrete variables.
- Channel capacity for noiseless channels.
- Source Coding Shannon Theorem
- Data Compression
- Kolmogorov Complexity
- Joint distributions
- Mutual Information
- Condicional Entropy
- Noise and cross comunicativos
- Error correcting codes
- Channel capacity for noisy-channels.
- The Noisy-Channel Coding Theorem
- Extension for the continuum domain.
- Information Theory on other knowledge domains
3 - Probability and Inference
4 - Neural Networks
Bibliography
- Information Theory, Inference, and Learning Algorithms, David J.C. MacKay - http://www.inference.org.uk/mackay/itila/book.html
- A Mathematical Theory of Communication, Claude Shannon - http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
- (some more later)
Teaching method
The lectures are used to present new topics, examples and adicional references when appropriate. The TP sessions will be mostly used to work on TP problems and partially to discuss small computer implementations. Some TP are dedicate to develop a small project.
The students performance evaluation includes two individual written tests and a small project.
Evaluation method
The students performance evaluation includes two individual written tests and a small project.
Final Grade = 35% Test1 + 35% Test2 + 30% Project
To successful conclude the following constraints are applied:
- Project >= 10;
- Test1 >= 8; Test2 >= 8;
- Average of Test1 and Test2 >= 10;
- Final Grade >= 10.
The students that get a Project >= 10 and do satisfy the constraints on the tests, may have an exam which grade will replace the tests in the final grade calculation