Logistic Regression and Maximum Entropy
Note for John Mount's "The Equivalence of Logistic Regression and Maximum Entropy Models" and explains that this proof is a special case of the general derivation proof of the maximum entropy model introduced in statistical learning methods
Conclusion
- Maximum entropy model is softmax classification
- Under the balanced conditions of the general linear model, the model mapping function that satisfies the maximum entropy condition is the softmax function
- In the book on Statistical Machine Learning methods, a maximum entropy model defined under the feature function is presented, which, along with softmax regression, belongs to the class of log-linear models
- When the feature function extends from a binary function to the feature value itself, the maximum entropy model becomes a softmax regression model
- The maximum entropy maximizes conditional entropy, not the entropy of conditional probabilities, nor the entropy of joint probabilities.