Python: Iterating moving Folders
I have over 52k files of csv files that I need to organize and would like to find an efficient way to do this through python or some other avenue.
Currently I have these folders,
2013_Q1
2013_Q2
2013_Q3
2013_Q4
2014_Q1 ...
and so on
Within the Quarter folder, i have another folder:
xxxx20130101_000500_csv
xxxx20130101_000500_xml
xxxx20130101_001000_csv
xxxx20130101_001000_xml
and so on..
within that folder I have the files:
xxxx20130101_000500_csv.csv
xxxx20130101_000500_xml.xml
xxxx20130101_001000_csv.csv
xxxx20130101_001000_xml.xml
respectively.
I want to go through all the quarter folders and extract only the .csv files from each sub-folder and organize them in a folder by their respective dates.
So within the 2013_Q1 folder, I would like to have
20130101
20130102...
and so on
And within that 20130101 folder would be
xxxx20130101_000500_csv.csv
xxxx20130101_001000_csv.csv
xxxx20130101_001500_csv.csv
Right now I have the python code:
import shutil
import os
os.chdir('C:\...\Test')
for f in os.listdir('MovingFolders'):
folderName = f[-19:-11]
if not os.path.exists(folderName):
os.mkdir(folderName)
shutil.copy(os.path.join('MovingFolders', f), folderName)
else:
shutil.copy(os.path.join('MovingFolders', f), folderName)
I'm fairly new at python and is still learning, so I'm a bit confused.
python csv shutil
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
add a comment |
I have over 52k files of csv files that I need to organize and would like to find an efficient way to do this through python or some other avenue.
Currently I have these folders,
2013_Q1
2013_Q2
2013_Q3
2013_Q4
2014_Q1 ...
and so on
Within the Quarter folder, i have another folder:
xxxx20130101_000500_csv
xxxx20130101_000500_xml
xxxx20130101_001000_csv
xxxx20130101_001000_xml
and so on..
within that folder I have the files:
xxxx20130101_000500_csv.csv
xxxx20130101_000500_xml.xml
xxxx20130101_001000_csv.csv
xxxx20130101_001000_xml.xml
respectively.
I want to go through all the quarter folders and extract only the .csv files from each sub-folder and organize them in a folder by their respective dates.
So within the 2013_Q1 folder, I would like to have
20130101
20130102...
and so on
And within that 20130101 folder would be
xxxx20130101_000500_csv.csv
xxxx20130101_001000_csv.csv
xxxx20130101_001500_csv.csv
Right now I have the python code:
import shutil
import os
os.chdir('C:\...\Test')
for f in os.listdir('MovingFolders'):
folderName = f[-19:-11]
if not os.path.exists(folderName):
os.mkdir(folderName)
shutil.copy(os.path.join('MovingFolders', f), folderName)
else:
shutil.copy(os.path.join('MovingFolders', f), folderName)
I'm fairly new at python and is still learning, so I'm a bit confused.
python csv shutil
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
1
Welcome to SO. It would help if you could provide what errors did you encounter...
– sophros
Nov 16 '18 at 21:00
I recommend you look at the python libraryglob. It will let you use a wildcard likeglob.glob("*.csv")which will return only files with a CSV file extension.
– ritlew
Nov 16 '18 at 21:01
Where/how did you come up with thef[-19:-11]? Seem very path dependent.
– martineau
Nov 16 '18 at 21:11
I wanted to auto generate folder name using filename. And that was the string index of the file name i wanted.
– Winne Loo
Nov 16 '18 at 21:22
I got this error, PermissionError: [Errno 13] Permission denied: 'MovingFolders\xxxxxx_20180401_000015_xml'
– Winne Loo
Nov 16 '18 at 21:39
add a comment |
I have over 52k files of csv files that I need to organize and would like to find an efficient way to do this through python or some other avenue.
Currently I have these folders,
2013_Q1
2013_Q2
2013_Q3
2013_Q4
2014_Q1 ...
and so on
Within the Quarter folder, i have another folder:
xxxx20130101_000500_csv
xxxx20130101_000500_xml
xxxx20130101_001000_csv
xxxx20130101_001000_xml
and so on..
within that folder I have the files:
xxxx20130101_000500_csv.csv
xxxx20130101_000500_xml.xml
xxxx20130101_001000_csv.csv
xxxx20130101_001000_xml.xml
respectively.
I want to go through all the quarter folders and extract only the .csv files from each sub-folder and organize them in a folder by their respective dates.
So within the 2013_Q1 folder, I would like to have
20130101
20130102...
and so on
And within that 20130101 folder would be
xxxx20130101_000500_csv.csv
xxxx20130101_001000_csv.csv
xxxx20130101_001500_csv.csv
Right now I have the python code:
import shutil
import os
os.chdir('C:\...\Test')
for f in os.listdir('MovingFolders'):
folderName = f[-19:-11]
if not os.path.exists(folderName):
os.mkdir(folderName)
shutil.copy(os.path.join('MovingFolders', f), folderName)
else:
shutil.copy(os.path.join('MovingFolders', f), folderName)
I'm fairly new at python and is still learning, so I'm a bit confused.
python csv shutil
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
I have over 52k files of csv files that I need to organize and would like to find an efficient way to do this through python or some other avenue.
Currently I have these folders,
2013_Q1
2013_Q2
2013_Q3
2013_Q4
2014_Q1 ...
and so on
Within the Quarter folder, i have another folder:
xxxx20130101_000500_csv
xxxx20130101_000500_xml
xxxx20130101_001000_csv
xxxx20130101_001000_xml
and so on..
within that folder I have the files:
xxxx20130101_000500_csv.csv
xxxx20130101_000500_xml.xml
xxxx20130101_001000_csv.csv
xxxx20130101_001000_xml.xml
respectively.
I want to go through all the quarter folders and extract only the .csv files from each sub-folder and organize them in a folder by their respective dates.
So within the 2013_Q1 folder, I would like to have
20130101
20130102...
and so on
And within that 20130101 folder would be
xxxx20130101_000500_csv.csv
xxxx20130101_001000_csv.csv
xxxx20130101_001500_csv.csv
Right now I have the python code:
import shutil
import os
os.chdir('C:\...\Test')
for f in os.listdir('MovingFolders'):
folderName = f[-19:-11]
if not os.path.exists(folderName):
os.mkdir(folderName)
shutil.copy(os.path.join('MovingFolders', f), folderName)
else:
shutil.copy(os.path.join('MovingFolders', f), folderName)
I'm fairly new at python and is still learning, so I'm a bit confused.
python csv shutil
python csv shutil
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
edited Nov 16 '18 at 23:10
Hemerson Tacon
1,0021316
1,0021316
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
asked Nov 16 '18 at 20:57
Winne LooWinne Loo
1
1
1
Welcome to SO. It would help if you could provide what errors did you encounter...
– sophros
Nov 16 '18 at 21:00
I recommend you look at the python libraryglob. It will let you use a wildcard likeglob.glob("*.csv")which will return only files with a CSV file extension.
– ritlew
Nov 16 '18 at 21:01
Where/how did you come up with thef[-19:-11]? Seem very path dependent.
– martineau
Nov 16 '18 at 21:11
I wanted to auto generate folder name using filename. And that was the string index of the file name i wanted.
– Winne Loo
Nov 16 '18 at 21:22
I got this error, PermissionError: [Errno 13] Permission denied: 'MovingFolders\xxxxxx_20180401_000015_xml'
– Winne Loo
Nov 16 '18 at 21:39
add a comment |
1
Welcome to SO. It would help if you could provide what errors did you encounter...
– sophros
Nov 16 '18 at 21:00
I recommend you look at the python libraryglob. It will let you use a wildcard likeglob.glob("*.csv")which will return only files with a CSV file extension.
– ritlew
Nov 16 '18 at 21:01
Where/how did you come up with thef[-19:-11]? Seem very path dependent.
– martineau
Nov 16 '18 at 21:11
I wanted to auto generate folder name using filename. And that was the string index of the file name i wanted.
– Winne Loo
Nov 16 '18 at 21:22
I got this error, PermissionError: [Errno 13] Permission denied: 'MovingFolders\xxxxxx_20180401_000015_xml'
– Winne Loo
Nov 16 '18 at 21:39
1
1
Welcome to SO. It would help if you could provide what errors did you encounter...
– sophros
Nov 16 '18 at 21:00
Welcome to SO. It would help if you could provide what errors did you encounter...
– sophros
Nov 16 '18 at 21:00
I recommend you look at the python library
glob. It will let you use a wildcard like glob.glob("*.csv") which will return only files with a CSV file extension.– ritlew
Nov 16 '18 at 21:01
I recommend you look at the python library
glob. It will let you use a wildcard like glob.glob("*.csv") which will return only files with a CSV file extension.– ritlew
Nov 16 '18 at 21:01
Where/how did you come up with the
f[-19:-11]? Seem very path dependent.– martineau
Nov 16 '18 at 21:11
Where/how did you come up with the
f[-19:-11]? Seem very path dependent.– martineau
Nov 16 '18 at 21:11
I wanted to auto generate folder name using filename. And that was the string index of the file name i wanted.
– Winne Loo
Nov 16 '18 at 21:22
I wanted to auto generate folder name using filename. And that was the string index of the file name i wanted.
– Winne Loo
Nov 16 '18 at 21:22
I got this error, PermissionError: [Errno 13] Permission denied: 'MovingFolders\xxxxxx_20180401_000015_xml'
– Winne Loo
Nov 16 '18 at 21:39
I got this error, PermissionError: [Errno 13] Permission denied: 'MovingFolders\xxxxxx_20180401_000015_xml'
– Winne Loo
Nov 16 '18 at 21:39
add a comment |
1 Answer
1
active
oldest
votes
For Python 3.5 and above:
There is a recursive feature in glob that you could use.
Here's what you do
import glob
loop over glob.glob('root/**/*.csv',recursive = True)
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53345270%2fpython-iterating-moving-folders%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
For Python 3.5 and above:
There is a recursive feature in glob that you could use.
Here's what you do
import glob
loop over glob.glob('root/**/*.csv',recursive = True)
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
add a comment |
For Python 3.5 and above:
There is a recursive feature in glob that you could use.
Here's what you do
import glob
loop over glob.glob('root/**/*.csv',recursive = True)
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
add a comment |
For Python 3.5 and above:
There is a recursive feature in glob that you could use.
Here's what you do
import glob
loop over glob.glob('root/**/*.csv',recursive = True)
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
For Python 3.5 and above:
There is a recursive feature in glob that you could use.
Here's what you do
import glob
loop over glob.glob('root/**/*.csv',recursive = True)
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
answered Nov 16 '18 at 21:15
Akash DuttaAkash Dutta
1
1
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53345270%2fpython-iterating-moving-folders%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Welcome to SO. It would help if you could provide what errors did you encounter...
– sophros
Nov 16 '18 at 21:00
I recommend you look at the python library
glob. It will let you use a wildcard likeglob.glob("*.csv")which will return only files with a CSV file extension.– ritlew
Nov 16 '18 at 21:01
Where/how did you come up with the
f[-19:-11]? Seem very path dependent.– martineau
Nov 16 '18 at 21:11
I wanted to auto generate folder name using filename. And that was the string index of the file name i wanted.
– Winne Loo
Nov 16 '18 at 21:22
I got this error, PermissionError: [Errno 13] Permission denied: 'MovingFolders\xxxxxx_20180401_000015_xml'
– Winne Loo
Nov 16 '18 at 21:39