What is the most efficient way to read heavy files(>~10Gb) using Node.JS(LTS)?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
What is the most efficient way to read heavy files(>~10Gb) using Node.JS(LTS)?
Essentially in today's world, I need to read the file content, parse each line to a known data-structure , perform certain validations, and push the data-structure into the database(SQL Server). I do it using C#(memory-mapped files). It works pretty well because I am able to read the file in chunks (in parallel).
I am planning to migrate the solution to Node(and MongoDB) for a business use-case.
Any leads/suggestions?
Environment:
I am using a 64-bit Windows OS, x64 based processor, 8Gigs of RAM
node.js file memory-mapped-files
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
add a comment |
What is the most efficient way to read heavy files(>~10Gb) using Node.JS(LTS)?
Essentially in today's world, I need to read the file content, parse each line to a known data-structure , perform certain validations, and push the data-structure into the database(SQL Server). I do it using C#(memory-mapped files). It works pretty well because I am able to read the file in chunks (in parallel).
I am planning to migrate the solution to Node(and MongoDB) for a business use-case.
Any leads/suggestions?
Environment:
I am using a 64-bit Windows OS, x64 based processor, 8Gigs of RAM
node.js file memory-mapped-files
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
add a comment |
What is the most efficient way to read heavy files(>~10Gb) using Node.JS(LTS)?
Essentially in today's world, I need to read the file content, parse each line to a known data-structure , perform certain validations, and push the data-structure into the database(SQL Server). I do it using C#(memory-mapped files). It works pretty well because I am able to read the file in chunks (in parallel).
I am planning to migrate the solution to Node(and MongoDB) for a business use-case.
Any leads/suggestions?
Environment:
I am using a 64-bit Windows OS, x64 based processor, 8Gigs of RAM
node.js file memory-mapped-files
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
What is the most efficient way to read heavy files(>~10Gb) using Node.JS(LTS)?
Essentially in today's world, I need to read the file content, parse each line to a known data-structure , perform certain validations, and push the data-structure into the database(SQL Server). I do it using C#(memory-mapped files). It works pretty well because I am able to read the file in chunks (in parallel).
I am planning to migrate the solution to Node(and MongoDB) for a business use-case.
Any leads/suggestions?
Environment:
I am using a 64-bit Windows OS, x64 based processor, 8Gigs of RAM
node.js file memory-mapped-files
node.js file memory-mapped-files
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
asked Nov 25 '18 at 4:54
Hanuma Hanuma
362
362
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
What you're looking for is usually referred to as streams in node.js.
You can read or write very large files with streams by processing portions of it.
Here are a few links that could help you to get started.
Parsing huge logfiles in Node.js - read in line-by-line
Using Node.js to Read Really, Really Large Datasets & Files
Read large text files in nodejs
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
1
Additionally... there are often standard filter streams and stream parsers for handling common file types.
– Brad
Nov 25 '18 at 5:07
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53464755%2fwhat-is-the-most-efficient-way-to-read-heavy-files10gb-using-node-jslts%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
What you're looking for is usually referred to as streams in node.js.
You can read or write very large files with streams by processing portions of it.
Here are a few links that could help you to get started.
Parsing huge logfiles in Node.js - read in line-by-line
Using Node.js to Read Really, Really Large Datasets & Files
Read large text files in nodejs
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
1
Additionally... there are often standard filter streams and stream parsers for handling common file types.
– Brad
Nov 25 '18 at 5:07
add a comment |
What you're looking for is usually referred to as streams in node.js.
You can read or write very large files with streams by processing portions of it.
Here are a few links that could help you to get started.
Parsing huge logfiles in Node.js - read in line-by-line
Using Node.js to Read Really, Really Large Datasets & Files
Read large text files in nodejs
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
1
Additionally... there are often standard filter streams and stream parsers for handling common file types.
– Brad
Nov 25 '18 at 5:07
add a comment |
What you're looking for is usually referred to as streams in node.js.
You can read or write very large files with streams by processing portions of it.
Here are a few links that could help you to get started.
Parsing huge logfiles in Node.js - read in line-by-line
Using Node.js to Read Really, Really Large Datasets & Files
Read large text files in nodejs
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
What you're looking for is usually referred to as streams in node.js.
You can read or write very large files with streams by processing portions of it.
Here are a few links that could help you to get started.
Parsing huge logfiles in Node.js - read in line-by-line
Using Node.js to Read Really, Really Large Datasets & Files
Read large text files in nodejs
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
answered Nov 25 '18 at 5:03
Dinesh PandiyanDinesh Pandiyan
2,79811028
2,79811028
1
Additionally... there are often standard filter streams and stream parsers for handling common file types.
– Brad
Nov 25 '18 at 5:07
add a comment |
1
Additionally... there are often standard filter streams and stream parsers for handling common file types.
– Brad
Nov 25 '18 at 5:07
1
1
Additionally... there are often standard filter streams and stream parsers for handling common file types.
– Brad
Nov 25 '18 at 5:07
Additionally... there are often standard filter streams and stream parsers for handling common file types.
– Brad
Nov 25 '18 at 5:07
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53464755%2fwhat-is-the-most-efficient-way-to-read-heavy-files10gb-using-node-jslts%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
