Python - load an in-memory ZipFile object as bytes












1















I have a script which creates a closed in-memory ZipFile object that I need to post as a bytestring (using requests); how do I do that? I have tried opening the file, which fails with "TypeError: expected str, bytes or os.PathLike object, not ZipFile"



The script works just fine if I write the ZipFile to a file and then open that file for the post data. However it will probably iterate over a couple million files, and that seems like a lot of temp files, and disk activity.



import io
import zipfile
from PIL import Image

z = io.BytesIO()
zfile = zipfile.ZipFile(z,"a")

zipdict = {}

img_loc = "D:/Images/seasons-3.jpg"
im_original = Image.open(img_loc)
imfmt = im_original.format
im = im_original.copy()
im_original.close()
im_out = io.BytesIO()
im.save(im_out,imfmt)
zfile.writestr("seasons-3.jpg",im_out.getvalue())
im_out.close()
zipdict['seasons-3']=zfile
zfile.close()


running with error:



Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
>>> zipdict['seasons-3']
<zipfile.ZipFile [closed]>
>>> pl_data = open(zipdict['seasons-3'])
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
pl_data = open(zipdict['seasons-3'])
TypeError: expected str, bytes or os.PathLike object, not ZipFile
>>>









share|improve this question




















  • 1





    Can you paste your code and error traceback?

    – Cheche
    Nov 20 '18 at 16:51






  • 2





    "closed in-memory ZipFile" - you're going to have to explain that some more. Did you wrap a ZipFile around a BytesIO or something?

    – user2357112
    Nov 20 '18 at 16:51











  • I'm not sure if it helps with an in-memory zip-file (never encountered one in the wild), but you can unzip a single file from an archive: stackoverflow.com/a/46423414/962190

    – Arne
    Nov 20 '18 at 16:56











  • @user2357112 that's pretty much exactly what I did. I created a ZipFile and used writestr to add a couple of BytesIO to the ZipFile. Then I added the ZipFile as value to a dict, with key as filename, and closed the ZipFile.

    – Tim Achee
    Nov 21 '18 at 7:13











  • @TimAchee: Nope, that still doesn't explain things. What, if anything, did you do to put the ZipFile itself in memory? What arguments did you pass to the ZipFile constructor?

    – user2357112
    Nov 21 '18 at 7:33


















1















I have a script which creates a closed in-memory ZipFile object that I need to post as a bytestring (using requests); how do I do that? I have tried opening the file, which fails with "TypeError: expected str, bytes or os.PathLike object, not ZipFile"



The script works just fine if I write the ZipFile to a file and then open that file for the post data. However it will probably iterate over a couple million files, and that seems like a lot of temp files, and disk activity.



import io
import zipfile
from PIL import Image

z = io.BytesIO()
zfile = zipfile.ZipFile(z,"a")

zipdict = {}

img_loc = "D:/Images/seasons-3.jpg"
im_original = Image.open(img_loc)
imfmt = im_original.format
im = im_original.copy()
im_original.close()
im_out = io.BytesIO()
im.save(im_out,imfmt)
zfile.writestr("seasons-3.jpg",im_out.getvalue())
im_out.close()
zipdict['seasons-3']=zfile
zfile.close()


running with error:



Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
>>> zipdict['seasons-3']
<zipfile.ZipFile [closed]>
>>> pl_data = open(zipdict['seasons-3'])
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
pl_data = open(zipdict['seasons-3'])
TypeError: expected str, bytes or os.PathLike object, not ZipFile
>>>









share|improve this question




















  • 1





    Can you paste your code and error traceback?

    – Cheche
    Nov 20 '18 at 16:51






  • 2





    "closed in-memory ZipFile" - you're going to have to explain that some more. Did you wrap a ZipFile around a BytesIO or something?

    – user2357112
    Nov 20 '18 at 16:51











  • I'm not sure if it helps with an in-memory zip-file (never encountered one in the wild), but you can unzip a single file from an archive: stackoverflow.com/a/46423414/962190

    – Arne
    Nov 20 '18 at 16:56











  • @user2357112 that's pretty much exactly what I did. I created a ZipFile and used writestr to add a couple of BytesIO to the ZipFile. Then I added the ZipFile as value to a dict, with key as filename, and closed the ZipFile.

    – Tim Achee
    Nov 21 '18 at 7:13











  • @TimAchee: Nope, that still doesn't explain things. What, if anything, did you do to put the ZipFile itself in memory? What arguments did you pass to the ZipFile constructor?

    – user2357112
    Nov 21 '18 at 7:33
















1












1








1








I have a script which creates a closed in-memory ZipFile object that I need to post as a bytestring (using requests); how do I do that? I have tried opening the file, which fails with "TypeError: expected str, bytes or os.PathLike object, not ZipFile"



The script works just fine if I write the ZipFile to a file and then open that file for the post data. However it will probably iterate over a couple million files, and that seems like a lot of temp files, and disk activity.



import io
import zipfile
from PIL import Image

z = io.BytesIO()
zfile = zipfile.ZipFile(z,"a")

zipdict = {}

img_loc = "D:/Images/seasons-3.jpg"
im_original = Image.open(img_loc)
imfmt = im_original.format
im = im_original.copy()
im_original.close()
im_out = io.BytesIO()
im.save(im_out,imfmt)
zfile.writestr("seasons-3.jpg",im_out.getvalue())
im_out.close()
zipdict['seasons-3']=zfile
zfile.close()


running with error:



Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
>>> zipdict['seasons-3']
<zipfile.ZipFile [closed]>
>>> pl_data = open(zipdict['seasons-3'])
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
pl_data = open(zipdict['seasons-3'])
TypeError: expected str, bytes or os.PathLike object, not ZipFile
>>>









share|improve this question
















I have a script which creates a closed in-memory ZipFile object that I need to post as a bytestring (using requests); how do I do that? I have tried opening the file, which fails with "TypeError: expected str, bytes or os.PathLike object, not ZipFile"



The script works just fine if I write the ZipFile to a file and then open that file for the post data. However it will probably iterate over a couple million files, and that seems like a lot of temp files, and disk activity.



import io
import zipfile
from PIL import Image

z = io.BytesIO()
zfile = zipfile.ZipFile(z,"a")

zipdict = {}

img_loc = "D:/Images/seasons-3.jpg"
im_original = Image.open(img_loc)
imfmt = im_original.format
im = im_original.copy()
im_original.close()
im_out = io.BytesIO()
im.save(im_out,imfmt)
zfile.writestr("seasons-3.jpg",im_out.getvalue())
im_out.close()
zipdict['seasons-3']=zfile
zfile.close()


running with error:



Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
>>> zipdict['seasons-3']
<zipfile.ZipFile [closed]>
>>> pl_data = open(zipdict['seasons-3'])
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
pl_data = open(zipdict['seasons-3'])
TypeError: expected str, bytes or os.PathLike object, not ZipFile
>>>






python python-requests zipfile bytesio






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 21 '18 at 8:19







Tim Achee

















asked Nov 20 '18 at 16:47









Tim AcheeTim Achee

122




122








  • 1





    Can you paste your code and error traceback?

    – Cheche
    Nov 20 '18 at 16:51






  • 2





    "closed in-memory ZipFile" - you're going to have to explain that some more. Did you wrap a ZipFile around a BytesIO or something?

    – user2357112
    Nov 20 '18 at 16:51











  • I'm not sure if it helps with an in-memory zip-file (never encountered one in the wild), but you can unzip a single file from an archive: stackoverflow.com/a/46423414/962190

    – Arne
    Nov 20 '18 at 16:56











  • @user2357112 that's pretty much exactly what I did. I created a ZipFile and used writestr to add a couple of BytesIO to the ZipFile. Then I added the ZipFile as value to a dict, with key as filename, and closed the ZipFile.

    – Tim Achee
    Nov 21 '18 at 7:13











  • @TimAchee: Nope, that still doesn't explain things. What, if anything, did you do to put the ZipFile itself in memory? What arguments did you pass to the ZipFile constructor?

    – user2357112
    Nov 21 '18 at 7:33
















  • 1





    Can you paste your code and error traceback?

    – Cheche
    Nov 20 '18 at 16:51






  • 2





    "closed in-memory ZipFile" - you're going to have to explain that some more. Did you wrap a ZipFile around a BytesIO or something?

    – user2357112
    Nov 20 '18 at 16:51











  • I'm not sure if it helps with an in-memory zip-file (never encountered one in the wild), but you can unzip a single file from an archive: stackoverflow.com/a/46423414/962190

    – Arne
    Nov 20 '18 at 16:56











  • @user2357112 that's pretty much exactly what I did. I created a ZipFile and used writestr to add a couple of BytesIO to the ZipFile. Then I added the ZipFile as value to a dict, with key as filename, and closed the ZipFile.

    – Tim Achee
    Nov 21 '18 at 7:13











  • @TimAchee: Nope, that still doesn't explain things. What, if anything, did you do to put the ZipFile itself in memory? What arguments did you pass to the ZipFile constructor?

    – user2357112
    Nov 21 '18 at 7:33










1




1





Can you paste your code and error traceback?

– Cheche
Nov 20 '18 at 16:51





Can you paste your code and error traceback?

– Cheche
Nov 20 '18 at 16:51




2




2





"closed in-memory ZipFile" - you're going to have to explain that some more. Did you wrap a ZipFile around a BytesIO or something?

– user2357112
Nov 20 '18 at 16:51





"closed in-memory ZipFile" - you're going to have to explain that some more. Did you wrap a ZipFile around a BytesIO or something?

– user2357112
Nov 20 '18 at 16:51













I'm not sure if it helps with an in-memory zip-file (never encountered one in the wild), but you can unzip a single file from an archive: stackoverflow.com/a/46423414/962190

– Arne
Nov 20 '18 at 16:56





I'm not sure if it helps with an in-memory zip-file (never encountered one in the wild), but you can unzip a single file from an archive: stackoverflow.com/a/46423414/962190

– Arne
Nov 20 '18 at 16:56













@user2357112 that's pretty much exactly what I did. I created a ZipFile and used writestr to add a couple of BytesIO to the ZipFile. Then I added the ZipFile as value to a dict, with key as filename, and closed the ZipFile.

– Tim Achee
Nov 21 '18 at 7:13





@user2357112 that's pretty much exactly what I did. I created a ZipFile and used writestr to add a couple of BytesIO to the ZipFile. Then I added the ZipFile as value to a dict, with key as filename, and closed the ZipFile.

– Tim Achee
Nov 21 '18 at 7:13













@TimAchee: Nope, that still doesn't explain things. What, if anything, did you do to put the ZipFile itself in memory? What arguments did you pass to the ZipFile constructor?

– user2357112
Nov 21 '18 at 7:33







@TimAchee: Nope, that still doesn't explain things. What, if anything, did you do to put the ZipFile itself in memory? What arguments did you pass to the ZipFile constructor?

– user2357112
Nov 21 '18 at 7:33














1 Answer
1






active

oldest

votes


















0














zfile is closed. It's useless to you. The thing you need to use now is z, the file-like object that was managing the underlying binary storage for the ZipFile.



You can use z.getvalue() to get a bytestring representing the contents of z, just like you did with im_out, or you can seek back to the beginning with z.seek(0) and use it with the parts of requests that take file-like objects.






share|improve this answer
























  • thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out.

    – Tim Achee
    Nov 21 '18 at 9:20











  • @TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object.

    – user2357112
    Nov 21 '18 at 19:21











  • thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file"

    – Tim Achee
    Nov 22 '18 at 7:35











  • @TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into z besides the zip contents. It's hard to tell from here.

    – user2357112
    Nov 22 '18 at 7:52











  • yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right?

    – Tim Achee
    Nov 22 '18 at 8:46











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53397725%2fpython-load-an-in-memory-zipfile-object-as-bytes%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














zfile is closed. It's useless to you. The thing you need to use now is z, the file-like object that was managing the underlying binary storage for the ZipFile.



You can use z.getvalue() to get a bytestring representing the contents of z, just like you did with im_out, or you can seek back to the beginning with z.seek(0) and use it with the parts of requests that take file-like objects.






share|improve this answer
























  • thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out.

    – Tim Achee
    Nov 21 '18 at 9:20











  • @TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object.

    – user2357112
    Nov 21 '18 at 19:21











  • thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file"

    – Tim Achee
    Nov 22 '18 at 7:35











  • @TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into z besides the zip contents. It's hard to tell from here.

    – user2357112
    Nov 22 '18 at 7:52











  • yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right?

    – Tim Achee
    Nov 22 '18 at 8:46
















0














zfile is closed. It's useless to you. The thing you need to use now is z, the file-like object that was managing the underlying binary storage for the ZipFile.



You can use z.getvalue() to get a bytestring representing the contents of z, just like you did with im_out, or you can seek back to the beginning with z.seek(0) and use it with the parts of requests that take file-like objects.






share|improve this answer
























  • thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out.

    – Tim Achee
    Nov 21 '18 at 9:20











  • @TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object.

    – user2357112
    Nov 21 '18 at 19:21











  • thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file"

    – Tim Achee
    Nov 22 '18 at 7:35











  • @TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into z besides the zip contents. It's hard to tell from here.

    – user2357112
    Nov 22 '18 at 7:52











  • yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right?

    – Tim Achee
    Nov 22 '18 at 8:46














0












0








0







zfile is closed. It's useless to you. The thing you need to use now is z, the file-like object that was managing the underlying binary storage for the ZipFile.



You can use z.getvalue() to get a bytestring representing the contents of z, just like you did with im_out, or you can seek back to the beginning with z.seek(0) and use it with the parts of requests that take file-like objects.






share|improve this answer













zfile is closed. It's useless to you. The thing you need to use now is z, the file-like object that was managing the underlying binary storage for the ZipFile.



You can use z.getvalue() to get a bytestring representing the contents of z, just like you did with im_out, or you can seek back to the beginning with z.seek(0) and use it with the parts of requests that take file-like objects.







share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 21 '18 at 8:57









user2357112user2357112

156k12168263




156k12168263













  • thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out.

    – Tim Achee
    Nov 21 '18 at 9:20











  • @TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object.

    – user2357112
    Nov 21 '18 at 19:21











  • thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file"

    – Tim Achee
    Nov 22 '18 at 7:35











  • @TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into z besides the zip contents. It's hard to tell from here.

    – user2357112
    Nov 22 '18 at 7:52











  • yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right?

    – Tim Achee
    Nov 22 '18 at 8:46



















  • thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out.

    – Tim Achee
    Nov 21 '18 at 9:20











  • @TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object.

    – user2357112
    Nov 21 '18 at 19:21











  • thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file"

    – Tim Achee
    Nov 22 '18 at 7:35











  • @TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into z besides the zip contents. It's hard to tell from here.

    – user2357112
    Nov 22 '18 at 7:52











  • yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right?

    – Tim Achee
    Nov 22 '18 at 8:46

















thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out.

– Tim Achee
Nov 21 '18 at 9:20





thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out.

– Tim Achee
Nov 21 '18 at 9:20













@TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object.

– user2357112
Nov 21 '18 at 19:21





@TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object.

– user2357112
Nov 21 '18 at 19:21













thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file"

– Tim Achee
Nov 22 '18 at 7:35





thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file"

– Tim Achee
Nov 22 '18 at 7:35













@TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into z besides the zip contents. It's hard to tell from here.

– user2357112
Nov 22 '18 at 7:52





@TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into z besides the zip contents. It's hard to tell from here.

– user2357112
Nov 22 '18 at 7:52













yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right?

– Tim Achee
Nov 22 '18 at 8:46





yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right?

– Tim Achee
Nov 22 '18 at 8:46




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53397725%2fpython-load-an-in-memory-zipfile-object-as-bytes%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







這個網誌中的熱門文章

Tangent Lines Diagram Along Smooth Curve

Yusuf al-Mu'taman ibn Hud

Zucchini