Out of Memory with MultiPart Post, Python 2.7 -



Out of Memory with MultiPart Post, Python 2.7 -

i have 300 mb file need upload, , current code isn't cutting it.

#---------------------------------------------------------------------------------- def _post_multipart(self, host, selector, fields, files, ssl=false,port=80, proxy_url=none,proxy_port=none): """ performs multi-post agol, portal, or ags inputs: host - string - root url (no http:// or https://) ex: www.arcgis.com selector - string - after host ex: /pwjussdojdp7sglj/arcgis/rest/services/gridindexfeatures/featureserver/0/1/addattachment fields - dictionary - additional parameters token , format info files - tuple array- tuple file name type, filename, total path ssl - alternative utilize ssl proxy_url - string - url proxy server proxy_port - interger - port value if not on port 80 output: json response dictionary useage: import urlparse url = "http://sampleserver3.arcgisonline.com/arcgis/rest/services/sanfrancisco/311incidents/featureserver/0/10261291" parsed_url = urlparse.urlparse(url) params = {"f":"json"} print _post_multipart(host=parsed_url.hostname, selector=parsed_url.path, files=files, fields=params ) """ content_type, body = self._encode_multipart_formdata(fields, files) headers = { 'content-type': content_type, 'content-length': str(len(body)) } if proxy_url: if ssl: h = httplib.httpsconnection(proxy_url, proxy_port) h.request('post', 'https://' + host + selector, body, headers) else: h = httplib.httpconnection(proxy_url, proxy_port) h.request('post', 'http://' + host + selector, body, headers) else: if ssl: h = httplib.httpsconnection(host,port) h.request('post', selector, body, headers) else: h = httplib.httpconnection(host,port) h.request('post', selector, body, headers) resp_data = h.getresponse().read() try: result = json.loads(resp_data) except: homecoming none if 'error' in result: if result['error']['message'] == 'request not made on ssl': homecoming self._post_multipart(host=host, selector=selector, fields=fields, files=files, ssl=true,port=port, proxy_url=proxy_url,proxy_port=proxy_port) homecoming result def _encode_multipart_formdata(self, fields, files): boundary = mimetools.choose_boundary() buf = stringio() (key, value) in fields.iteritems(): buf.write('--%s\r\n' % boundary) buf.write('content-disposition: form-data; name="%s"' % key) buf.write('\r\n\r\n' + self._tostr(value) + '\r\n') (key, filepath, filename) in files: if os.path.isfile(filepath): buf.write('--%s\r\n' % boundary) buf.write('content-disposition: form-data; name="%s"; filename="%s"\r\n' % (key, filename)) buf.write('content-type: %s\r\n' % (self._get_content_type3(filename))) file = open(filepath, "rb") try: buf.write('\r\n' + file.read() + '\r\n') finally: file.close() buf.write('--' + boundary + '--\r\n\r\n') buf = buf.getvalue() content_type = 'multipart/form-data; boundary=%s' % boundary homecoming content_type, buf

i cannot utilize requests module, , must utilize standard libraries urllib2, urllib, etc.. python 2.7.x.

is there way load 300 mb files site without pushing whole thing memory?

update:

so switched requests, , get: missingschema: invalid url u'www.arcgis.com/sharing/rest/content/users//additem?': no schema supplied. perhaps meant http://www.arcgis.com/sharing/rest/content/users//additem??

what mean?

i provide fields request.post() such:

#---------------------------------------------------------------------------------- def _post_big_files(self, host, selector, fields, files, ssl=false,port=80, proxy_url=none,proxy_port=none): import sys sys.path.insert(1,os.path.dirname(__file__)) requests_toolbelt import multipartencoder import requests if proxy_url not none: proxydict = { "http" : "%s:%s" % (proxy_url, proxy_port), "https" : "%s:%s" % (proxy_url, proxy_port) } else: proxydict = {} k,v in fields.iteritems(): print k,v fields[k] = json.dumps(v) key, filepath, filename in files: fields[key] = ('filename', open(filepath, 'rb'), self._get_content_type3(filepath)) m = multipartencoder( fields=fields) print host + selector r = requests.post(host + selector , data=m, headers={'content-type': m.content_type}) print r

i followed illustration in help documentation both request , toolbelt. ideas why breaking?

thank you,

python python-2.7 urllib2 urllib multipart-form

Comments

Post a Comment

Popular posts from this blog

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* -

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* - i've inherited doc publishing process takes docbook , builds html , pdf output, using apache fop both. project had been started, never completed. lots of tweaking, i've able of doc sets build (20+), except handful, , fail severe code 1 of these 2 conditions: "fo:flow" missing kid elements. required content model: marker* "fo:block" not valid kid of "fo:root" the xml output comes out of docbook , docbook-fop checks out well-formed. the fop error in 1 book, "endecagloss", example, points end of string: <fo:flow flow-name="xsl-region-body" start-indent="4pc" end-indent="0pt"/> here’s log output in case: [java] severe: javax.xml.transform.transformerexception: file:/scratch/publishing/hudson/jobs/build-main-endeca-documentation/workspace/serverdoc/w...
Read more

mediawiki - How do I insert tables inside infoboxes on Wikia pages? -

mediawiki - How do I insert tables inside infoboxes on Wikia pages? - i want insert tables in infoboxes. example, have proteinbox i've developed using infobox template (http://health-and-medicine.wikia.com/wiki/template:proteinbox), , fasta field big fit comfortably in proteinbox. hence collapsible table within box. this tried: {{infobox | row 1 = {| class = "mw-collapsible mw-collapsed wikitable" |- | <!-- table content --> |} }} your problem is, putting nested table within template. pipe characters ( | ) in table syntax collides pipe usage in templates. the commonly used hack around create template called template:! , containing pipe character, , utilize when need set tables, parser functions or other stuff using pipe characters, within templates. table (with every | replaced {{!}} ): {{{!}} {{!}}- {{!}} a1 {{!}} b1 {{!}}- {{!}} a2 {{!}} b2 {{!}}} ...the equivalent of {| |- | a1 | b1 |- | a2 | b2 |} furthermore, hav...
Read more

Local Service User Logged into Windows -

Local Service User Logged into Windows - i created windows service tracks app server user connected through load balancer. it retrieves user logged windows on machine, cross references active directory (ad) business relationship homecoming first name, lastly name , e-mail (for notification through website created project) there 1 scheme on network users remote system, vista system, reason returning values "local service" logged machine, it's weird , security guys impossible "local service" business relationship logged windows. here how user logged windows managementscope ms = new managementscope("\\\\.\\root\\cimv2"); objectquery query = new objectquery("select * win32_computersystem"); managementobjectsearcher searcher = new managementobjectsearcher(ms, query); foreach (managementobject mo in searcher.get()) { _username = mo["username"].tostring(); } // remove domain part username string[] usernameparts = _...
Read more