ThmDex – An index of mathematical definitions, results, and conjectures.
Gibbs' inequality for simple relative entropy
Formulation 0
Let $X, Y \in \text{Random}(\mathcal{X})$ each be a D5723: Simple random variable.
Let $\log_a$ be the D866: Standard real logarithm function in base $a \in (0, \infty) \setminus \{ 1 \}$.
Then
(1) \begin{equation} - \sum_{x \in \mathcal{X}} \mathbb{P}(X = x) \log_a \mathbb{P}(X = x) \leq - \sum_{x \in \mathcal{X}} \mathbb{P}(X = x) \log_a \mathbb{P}(Y = x) \end{equation}
(2) \begin{equation} - \sum_{x \in \mathcal{X}} \mathbb{P}(X = x) \log_a \mathbb{P}(X = x) = - \sum_{x \in \mathcal{X}} \mathbb{P}(X = x) \log_a \mathbb{P}(Y = x) \quad \iff \quad \forall \, x \in \mathcal{X} : \mathbb{P}(X = x) = \mathbb{P}(Y = x) \end{equation}
Formulation 1
Let $X, Y \in \text{Random}(\mathcal{X})$ each be a D5723: Simple random variable such that
(i) \begin{equation} \forall \, x \in \mathcal{X} : p(x) : = \mathbb{P}(X = x) \end{equation}
(ii) \begin{equation} \forall \, x \in \mathcal{X} : q(x) : = \mathbb{P}(Y = x) \end{equation}
Let $\log_a$ be the D866: Standard real logarithm function in base $a \in (0, \infty) \setminus \{ 1 \}$.
Then
(1) \begin{equation} - \sum_{x \in \mathcal{X}} p(x) \log_a p(x) \leq - \sum_{x \in \mathcal{X}} p(x) \log_a q(x) \end{equation}
(2) \begin{equation} - \sum_{x \in \mathcal{X}} p(x) \log_a p(x) = - \sum_{x \in \mathcal{X}} p(x) \log_a q(x) \quad \iff \quad \forall \, x \in \mathcal{X} : p(x) = q(x) \end{equation}
Formulation 2
Let $p_1, q_1, p_2, q_2, \, \ldots, \, p_N, q_N \in (0, 1]$ each be a D993: Real number such that
(i) \begin{equation} \sum_{n = 1}^N p_n = 1 \end{equation}
(ii) \begin{equation} \sum_{n = 1}^N q_n = 1 \end{equation}
Let $\log_a$ be the D866: Standard real logarithm function in base $a \in (0, \infty) \setminus \{ 1 \}$.
Then
(1) \begin{equation} - \sum_{n = 1}^N p_n \log_a p_n \leq - \sum_{n = 1}^N p_n \log_a q_n \end{equation}
(2) \begin{equation} - \sum_{n = 1}^N p_n \log_a p_n = - \sum_{n = 1}^N p_n \log_a q_n \quad \iff \quad \forall \, n \in \{ 1, 2, \, \ldots, \, N \} : p_n = q_n \end{equation}
Subresults
▶ R5647: Gibb's inequality for simple entropy