Is it Normal for a Neural Network Loss to Increase after being trained on an example?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I am currently testing an LSTM network. I print the loss of its prediction on a training example before back-propagation and after back-propagation. It would make sense that the after loss should always be less than the before loss because the network was just trained on that example.
However, I am noticing that around the 100th training example, the network begins to give a more inaccurate prediction after back-propagation than before back-propagating on a training example.
Is a network expected to always have the before loss be higher than the after loss? If so, are there any reasons this happens?
To be clear, for the first hundred examples, the network seems to be training correctly and doing just fine.
python machine-learning neural-network lstm recurrent-neural-network
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
add a comment |
I am currently testing an LSTM network. I print the loss of its prediction on a training example before back-propagation and after back-propagation. It would make sense that the after loss should always be less than the before loss because the network was just trained on that example.
However, I am noticing that around the 100th training example, the network begins to give a more inaccurate prediction after back-propagation than before back-propagating on a training example.
Is a network expected to always have the before loss be higher than the after loss? If so, are there any reasons this happens?
To be clear, for the first hundred examples, the network seems to be training correctly and doing just fine.
python machine-learning neural-network lstm recurrent-neural-network
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
Did you try decreasing your learning rate?
– cheersmate
Nov 26 '18 at 8:23
add a comment |
I am currently testing an LSTM network. I print the loss of its prediction on a training example before back-propagation and after back-propagation. It would make sense that the after loss should always be less than the before loss because the network was just trained on that example.
However, I am noticing that around the 100th training example, the network begins to give a more inaccurate prediction after back-propagation than before back-propagating on a training example.
Is a network expected to always have the before loss be higher than the after loss? If so, are there any reasons this happens?
To be clear, for the first hundred examples, the network seems to be training correctly and doing just fine.
python machine-learning neural-network lstm recurrent-neural-network
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
I am currently testing an LSTM network. I print the loss of its prediction on a training example before back-propagation and after back-propagation. It would make sense that the after loss should always be less than the before loss because the network was just trained on that example.
However, I am noticing that around the 100th training example, the network begins to give a more inaccurate prediction after back-propagation than before back-propagating on a training example.
Is a network expected to always have the before loss be higher than the after loss? If so, are there any reasons this happens?
To be clear, for the first hundred examples, the network seems to be training correctly and doing just fine.
python machine-learning neural-network lstm recurrent-neural-network
python machine-learning neural-network lstm recurrent-neural-network
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
asked Nov 25 '18 at 5:33
Rehaan AhmadRehaan Ahmad
452317
452317
Did you try decreasing your learning rate?
– cheersmate
Nov 26 '18 at 8:23
add a comment |
Did you try decreasing your learning rate?
– cheersmate
Nov 26 '18 at 8:23
Did you try decreasing your learning rate?
– cheersmate
Nov 26 '18 at 8:23
Did you try decreasing your learning rate?
– cheersmate
Nov 26 '18 at 8:23
add a comment |
1 Answer
1
active
oldest
votes
Is your dataset shuffled?
Otherwise it could be the case that it was predicting one class for the first 99 examples.
If not then LSTM can be tricky to train. Try changing hyper parameters and also I would recommend starting with SimpleRNN, GRU and then LSTM as sometimes a simple network might just do the trick.
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53464933%2fis-it-normal-for-a-neural-network-loss-to-increase-after-being-trained-on-an-exa%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is your dataset shuffled?
Otherwise it could be the case that it was predicting one class for the first 99 examples.
If not then LSTM can be tricky to train. Try changing hyper parameters and also I would recommend starting with SimpleRNN, GRU and then LSTM as sometimes a simple network might just do the trick.
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
add a comment |
Is your dataset shuffled?
Otherwise it could be the case that it was predicting one class for the first 99 examples.
If not then LSTM can be tricky to train. Try changing hyper parameters and also I would recommend starting with SimpleRNN, GRU and then LSTM as sometimes a simple network might just do the trick.
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
add a comment |
Is your dataset shuffled?
Otherwise it could be the case that it was predicting one class for the first 99 examples.
If not then LSTM can be tricky to train. Try changing hyper parameters and also I would recommend starting with SimpleRNN, GRU and then LSTM as sometimes a simple network might just do the trick.
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
Is your dataset shuffled?
Otherwise it could be the case that it was predicting one class for the first 99 examples.
If not then LSTM can be tricky to train. Try changing hyper parameters and also I would recommend starting with SimpleRNN, GRU and then LSTM as sometimes a simple network might just do the trick.
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
answered Nov 25 '18 at 8:39
VitrioilVitrioil
8915
8915
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53464933%2fis-it-normal-for-a-neural-network-loss-to-increase-after-being-trained-on-an-exa%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Did you try decreasing your learning rate?
– cheersmate
Nov 26 '18 at 8:23