Information Theory and Coding

Higher education teachers: Dobrišek Simon

Subject description

Prerequisites:

• Completed undergraduate study programme in the field of electrical engineering or related engineering or natural and mathematical sciences.
• Enrolment in the 1st year of the Master’s study programme for Electrical Engineering (2nd cycle).
• Basic knowledge of mathematical analysis, linear algebra, probability theory and computer programming.

Content (Syllabus outline):

• Introduction: definition of information, code, coding and communication system.
• Entropy: discrete random systems, entropy of a discrete random variable, entropy of a continuous random variable.
• Information: average information, mutual information of two discrete variables, mutual information of two continuous variables.
• Discrete information sources: entropy of a stationary source, ergodic stationary sources, memoryless sources, sources with memory, and source redundancy.
• Information source coding: fixed-length and variable-length coding, Kraft – McMillan inequality, Huffman code, arithmetic code, LZW code.
• Secrecy coding: cryptosystems with a secret key, DES and AES cryptosystems, cryptosystems with a public key, RSA cryptosystem, digital signature.
• Communication channels I: continuous communication channels, the capacity of a continuous communication channel, discrete communication channels, the capacity of a discrete communication channel.
• Communication channels II: error-detecting codes, error-correcting codes, optimal decoding, Shannon theorem of secure coding, inversion of Shannon theorem.
• Channel coding: linear block codes, cyclic codes, Hamming codes, Golay codes, the Reed-Muller codes, convolutional codes, the Viterbi algorithm, Turbo codes.

Objectives and competences:

To provide students with an understanding of the most important concepts and methods from information theory, source coding, cryptography, and channel coding.

Intended learning outcomes:

• Knowledge and understanding: After completing this course the student will be able to demonstrate a knowledge and understanding of the: construction of the mathematical models used to describe the main phenomena in technical communication systems,
• information measure and other parameters used for a quantitative description of communication and information systems,information coding and cryptography.
• The use of knowledge: The student will be able to use the acquired knowledge to construct information encoders, in e-commerce as well to manage digital certificates for digital-identity management and encrypting messages. The student will be able to critically evaluate the consistency between the acquired knowledge and the application of the concepts of information theory, coding and cryptography in the fields of telecommun­ications, automation, control, and cybernetics.
• Transferable skills: the use of literature and other resources in the fields of information theory and coding; the use of development tools and environments for computer programming (writing computer programs in different programming languages, such as C/C++, C#, Java, Python, or using the Matlab development environment); problem solving: problem analysis, algorithm design, implementation and testing of a program; group work: the organisation and management of groups, active participation in groups.

Learning and teaching methods:

• lectures,
• laboratory exercises and projects,
• coursework.

Study materials

Readings:

1. T. M. Cover, J. A. Thomas: Elements of Information Theory, Wiley-Interscience, New York, 2006.
2. Roberto Togneri, Christopher J. S. deSilva: Fundamentals of information Theory and Coding Design, Chapman & Hall / CRC, 2002.
3. Douglas R. Stinson: Cryptography, Theory and Practice, Chapman & Hall / CRC, 2002.