Is it Normal for a Neural Network Loss to Increase after being trained on an example?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
0
I am currently testing an LSTM network. I print the loss of its prediction on a training example before back-propagation and after back-propagation. It would make sense that the after loss should always be less than the before loss because the network was just trained on that example. However, I am noticing that around the 100th training example, the network begins to give a more inaccurate prediction after back-propagation than before back-propagating on a training example. Is a network expected to always have the before loss be higher than the after loss? If so, are there any reasons this happens? To be clear, for the firs...
