Why use a confusion matrix?
To compare what Luminoso’s classifiers detected vs. what the true label was (the label on the training data)
To be able to see which labels are having overlap, and may be causing ‘confusion’ in the classifier.
To fix those areas of confusion, so that the classifier is in a better state to push into production
Reading a confusion matrix
An example of how to read a column in the above screenshot of a confusion matrix:
In D2 you see, ‘local host installs’.
This D column is to illustrate how the classification for the label ‘local host installs’ performed.
You can see that out of 304 testing documents (B8), 268 (B3) were correctly classified.
4 documents were classified with the label of ‘installation’, which differed from the training data tag of ‘local host install”.
Determining correction needs
Generally, we recommend inspecting and potentially correcting a label if the accuracy is below 80%.
This accuracy is calculated by taking the correct number, and dividing by the total number, for example in the above screenshot (B3/B8=accuracy)
Also, if there is an incorrect cell that is taking a large portion of the total incorrect classifications, this should be investigated to see why this large overlap/confusion is existing. A ‘large portion’ is not an exact threshold, but typically 10-20% of the total classifications is a good rule of thumb.