Information theory

Fall 2017 at ENS Lyon

General information

This course is an introduction to information theory and error correcting codes from the computer science point of view.


For some online resources on probability (for computer science), you can look at these notes or these notes (starting at Lecture 10).


The grading scheme will take into account


Date Topic References
Tue Sep 12th 10:15-12:15 Admin. Overview of the fundamental problem of noisy channel coding. Definition of entropy.
Ch 1 of [Mac], Ch 2 of [CT], Shannon's paper
Wed Sep 13th 15:45-17:45 Conditional entropy, mutual information, relative entropy. Data compression, variable-length lossless compressor.
Ch 8.1 of [PW], Ch 4 of [Mac], Ch 2,3,5 of [CT]
Fri Sep 22nd 9:00-12:00 Continuing variable-length coding. Prefix codes, Kraft's inequality, H(X) < L(C) < H(X)+1.
Fixed-length compression, general bounds in terms of the tail probabilities for the surprisal h_X(X).
Special case of memoryless sources: Shannon's source coding theorem.
Ch 8.2 of [PW], Ch 5,6 of [Mac], Ch 5 of [CT], Ch 9.1 of [PW]
Fri Oct 13th 9:00-12:00 Universal compression: arithmetic coding and Lempel-Ziv.
Channel coding: Information capacity of a channel. Additivity of information capacity for independent channels. Converse bounds
Ch 16.1 of [PW], Ch 6 of [Mac], Ch 16.3 [PW], Ch 7 of [CT].
Fri Oct 27rd 9:00-12:00 Achievability bounds. Shannon noisy coding theorem.
Ch 16.4 [PW], Ch 7 of [CT]
Fri Nov 10th 10:15-12:15 Midterm exam