A Measure-Theoretic Perspective on Multivariate Information with Applications to Data Science

by Kyle Schick-Poland
Doctoral thesis
Date of Examination:2024-10-23
Date of issue:2025-09-17
Advisor:Prof. Dr. Michael Wibral
Referee:Prof. Dr. Fred Wolf
Referee:Prof. Dr. Anja Sturm
Referee:Prof. Dr. Jens Niemeyer
Referee:Prof. Dr. Stefan Klumpp
Referee:Prof. Dr. Jörg Enderlein
Persistent Address: http://dx.doi.org/10.53846/goediss-11518

 

 

Files in this item

Name:PhD_thesis.pdf
Size:2.30Mb
Format:PDF

The following license files are associated with this item:

Abstract

English

This dissertation establishes a measure-theoretic foundation for information theory by formulating entropy, mutual information, and related quantities on a common type of probability spaces. Using Radon–Nikodym derivatives of induced probability measures, and Lebesgue integration, it extends Shannon’s framework beyond discrete variables to continuous and mixed distributions in a mathematically consistent manner. The main contribution is the introduction of Partial Information Decomposition (PID) within this framework. While continuous PID measures exist, they suffer from limited generality. Here, unique, redundant, and synergistic information are defined as functionals on measurable spaces, expressed via Radon–Nikodym derivatives of conditional probability measures with respect to product measures on σ-algebras. This establishes a PID based on redundancy rigorously across arbitrary measurable spaces. The measure-theoretic extension of TE is expressed in terms of conditional Radon–Nikodym derivatives on product σ-algebras, yielding a generalized quantification of directional information flow in stochastic processes. Building on this, new k-nearest neighbour estimators are derived for PID and Transfer Entropy (TE). These estimators extend the Kraskov–Stögbauer–Grassberger (KSG) method to assess the respective information terms in continuous and mixed distributions, with empirical proofs of convergence relying on the law of large numbers. By unifying discrete, continuous, and mixed settings under a σ-algebraic formalism and deriving provably convergent estimators, this work advances the mathematical foundations of information decomposition and informative measures, providing a rigorous framework for future developments in modern information theory.
Keywords: Information Theory; Partial Information Decomposition; Transfer Entropy; Estimation; Measure Theory
 
Statistik