python - How to write to cloudstorage using task chaining, The blobstore Files API did a nice job -



python - How to write to cloudstorage using task chaining, The blobstore Files API did a nice job -

using blobstore file api, write very big blobfiles:

create blobfile read info datastore , write (append) blobstore file pass datastore page cursor , blobstore file (next) task ....use other tasks same purpose and finalize blobstore file

now gae gcs client cannot append , finalize. how write big files gcs without compose. compose not part of gcs client. files api still works fine, has been deprecated.

below blobstore solution using task chaining:

class blobdata(webapp2.requesthandler): def post(self): page = int(self.request.get('page', default_value='0')) info = data.get_data(.....) blob_file = self.request.get('blobfile', default_value='none') if blob_file == 'none': file_name = files.blobstore.create(mime_type='text/...', _blobinfo_uploaded_filename='data....txt') else: data.with_cursor(self.request.get('cursor')) count = 0 # page lines counter files.open(blob_file, 'a') f: each in data.fetch(page): f.write(each) count += 1 if count >= page: cursor = data.cursor() taskqueue.add(url='/blobdata', queue_name='blobdata', countdown=10, method='post', params={'page': page, 'cursor': cursor, 'blobfile': blob_file}, headers={'x-appengine-failfast': 'true'}) else: # no info left files.finalize(blob_file)

in java client, can serialize reading channel (the equivalent of buffer in python client), , pass task go on writing in same file. see the java doc more info :

a readable byte channel reading info google cloud storage. implementations of class may buffer info internally cut down remote calls.

this class serializable, allows reading part of file, serializing gcsinputchannel deserializing it, , continuing read same file same position.

i not know if buffers returned python gcs client serializable, did not find info in doc, might worth checking.

if that's not possible, utilize composition. not worry fact composition not available in gcs client, can utilize standard cloud storage api library app engine. api documentation has compose illustration in python here. looks :

composite_object_resource = { 'contenttype': 'text/plain', # required 'contentlanguage': 'en', 'metadata': {'my-key': 'my-value'}, } compose_req_body = { 'sourceobjects': [ {'name': source_object_name_1, 'objectpreconditions': {'ifgenerationmatch': source_generation_1}}, {'name': source_object_name_2, 'objectpreconditions': {'ifgenerationmatch': source_generation_2}}], 'destination': composite_object_resource } req = client.objects().compose( destinationbucket=bucket_name, destinationobject=composite_object_name, body=compose_req_body) resp = req.execute() print json.dumps(resp, indent=2)

python google-app-engine python-2.7 google-cloud-storage blobstore

Comments

Post a Comment

Popular posts from this blog

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* -

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* - i've inherited doc publishing process takes docbook , builds html , pdf output, using apache fop both. project had been started, never completed. lots of tweaking, i've able of doc sets build (20+), except handful, , fail severe code 1 of these 2 conditions: "fo:flow" missing kid elements. required content model: marker* "fo:block" not valid kid of "fo:root" the xml output comes out of docbook , docbook-fop checks out well-formed. the fop error in 1 book, "endecagloss", example, points end of string: <fo:flow flow-name="xsl-region-body" start-indent="4pc" end-indent="0pt"/> here’s log output in case: [java] severe: javax.xml.transform.transformerexception: file:/scratch/publishing/hudson/jobs/build-main-endeca-documentation/workspace/serverdoc/w...
Read more

mediawiki - How do I insert tables inside infoboxes on Wikia pages? -

mediawiki - How do I insert tables inside infoboxes on Wikia pages? - i want insert tables in infoboxes. example, have proteinbox i've developed using infobox template (http://health-and-medicine.wikia.com/wiki/template:proteinbox), , fasta field big fit comfortably in proteinbox. hence collapsible table within box. this tried: {{infobox | row 1 = {| class = "mw-collapsible mw-collapsed wikitable" |- | <!-- table content --> |} }} your problem is, putting nested table within template. pipe characters ( | ) in table syntax collides pipe usage in templates. the commonly used hack around create template called template:! , containing pipe character, , utilize when need set tables, parser functions or other stuff using pipe characters, within templates. table (with every | replaced {{!}} ): {{{!}} {{!}}- {{!}} a1 {{!}} b1 {{!}}- {{!}} a2 {{!}} b2 {{!}}} ...the equivalent of {| |- | a1 | b1 |- | a2 | b2 |} furthermore, hav...
Read more

Local Service User Logged into Windows -

Local Service User Logged into Windows - i created windows service tracks app server user connected through load balancer. it retrieves user logged windows on machine, cross references active directory (ad) business relationship homecoming first name, lastly name , e-mail (for notification through website created project) there 1 scheme on network users remote system, vista system, reason returning values "local service" logged machine, it's weird , security guys impossible "local service" business relationship logged windows. here how user logged windows managementscope ms = new managementscope("\\\\.\\root\\cimv2"); objectquery query = new objectquery("select * win32_computersystem"); managementobjectsearcher searcher = new managementobjectsearcher(ms, query); foreach (managementobject mo in searcher.get()) { _username = mo["username"].tostring(); } // remove domain part username string[] usernameparts = _...
Read more