python - How to increase update performence with addToSet -



python - How to increase update performence with addToSet -

i have collection on 600000 records. records have construction below (from js console)

> db.invindex.find({_id: 'katar'}).pretty() { "_id" : "katar", "place" : [ { "index" : [ 1 ], "pageid" : objectid("53fd177620ba27591133387b") }, { "index" : [ 396 ], "pageid" : objectid("53dc621420ba2708ea53c0f5") }, { "index" : [ 4 ], "pageid" : objectid("53f6e20c20ba271058d5c70c") }, { "index" : [ 18, 41 ], "pageid" : objectid("53f6e20220ba271055d5c705") }, { "index" : [ 3 ], "pageid" : objectid("53f6e20420ba27105bd5c702") }, { "pageid" : objectid("53f6e25220ba271079318d44"), "index" : [ 21 ] }, { "pageid" : objectid("53f6e43220ba271061d5c81a"), "index" : [ 2 ] }, { "pageid" : objectid("53f6e43320ba271061d5c81b"), "index" : [ 2, 19 ] }, { "pageid" : objectid("53f6e42f20ba27105bd5c7ff"), "index" : [ 23 ] }, { "pageid" : objectid("53f6e4dd20ba27105bd5c860"), "index" : [ 8, 24 ] }, { "pageid" : objectid("53f6e63a20ba271076318ed4"), "index" : [ 6, 18 ] }, { "pageid" : objectid("53f6e63b20ba270ffd4533ea"), "index" : [ 2 ] }, { "pageid" : objectid("53f6e65720ba270ffd4533f6"), "index" : [ 428 ] }, { "pageid" : objectid("53f6eacb20ba271073319162"), "index" : [ 22 ] }, { "pageid" : objectid("53f6ed5d20ba271003453786"), "index" : [ 304 ] } ] }

the place field updated frequent. want add together new record on place sub-doc. below python procedure of how have accomplished it.

def update(doc, pos_list): add_to_set = {} add_to_set['place'] = {} add_to_set['place']['pageid'] = doc._id add_to_set['place']['index'] = pos_list # eg, pos_list = [1,2,3] info = self.collection.find_one({'_id': _id}) # check if 'pageid' there in record. # if not, update self.collection.update( {'_id': _id}, {'$addtoset': add_to_set}, upsert=upsert, w=1 )

but update seems snail-slow! avr 1.3 sec per update. i'm running 10 kid process boost speed. here mongostat

insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netin netout conn time *0 25 19 *0 0 1|0 0 68.1g 137g 691m 130 spider:65.1% 0 1|0 6|0 5k 63m 37 20:14:30 *0 2 1 *0 0 1|0 0 68.1g 137g 629m 194 spider:10.8% 0 0|0 2|0 647b 8m 37 20:14:31 *0 5 8 *0 0 1|0 0 68.1g 137g 650m 27 spider:71.9% 0 6|0 1|0 1k 4m 37 20:14:32 *0 15 15 *0 0 1|0 0 68.1g 137g 654m 105 spider:52.4% 0 7|0 0|1 4k 32m 37 20:14:34 *0 9 12 *0 0 1|0 0 68.1g 137g 666m 38 spider:53.9% 0 11|0 0|1 2k 7m 37 20:14:35 *0 16 13 *0 0 1|0 0 68.1g 137g 633m 222 spider:46.6% 0 2|0 1|0 4k 40m 37 20:14:36 *0 10 11 *0 0 1|0 0 68.1g 137g 666m 103 spider:39.5% 0 6|0 0|1 2k 22m 37 20:14:38 *0 13 15 *0 0 1|0 0 68.1g 137g 655m 20 spider:90.0% 0 14|0 0|1 3k 1m 37 20:14:39 *0 18 17 *0 0 1|0 0 68.1g 137g 672m 179 spider:48.5% 0 0|0 5|0 5k 47m 37 20:14:40 *0 12 15 *0 0 1|0 0 68.1g 137g 661m 119 spider:40.2% 0 2|0 4|1 3k 15m 37 20:14:41 insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netin netout conn time *0 9 8 *0 0 1|0 0 68.1g 137g 682m 109 spider:27.5% 0 6|0 0|1 2k 32m 37 20:14:43 *0 19 21 *0 0 1|0 0 68.1g 137g 659m 147 spider:85.6% 0 6|0 2|0 5k 20m 37 20:14:44 *0 15 20 *0 0 1|0 0 68.1g 137g 684m 137 spider:63.1% 0 9|0 1|0 4k 12m 37 20:14:45 *0 18 13 *0 0 1|0 0 68.1g 137g 634m 197 spider:39.0% 0 0|0 4|0 5k 51m 37 20:14:46 *0 6 6 *0 0 1|0 0 68.1g 137g 638m 209 spider:13.9% 0 0|0 2|0 1k 3m 37 20:14:47 *0 4 9 *0 0 1|0 0 68.1g 137g 641m 121 spider:37.4% 0 13|0 0|1 866b 3k 37 20:14:49 *0 19 15 *0 0 1|0 0 68.1g 137g 618m 141 spider:50.7% 0 0|0 4|0 5k 58m 37 20:14:50 *0 2 3 *0 0 1|0 0 68.1g 137g 624m 181 spider:15.4% 0 6|0 0|1 528b 207k 37 20:14:51

my index _id field. have 4 gb of ram. now, questions are

is normal speed? how can boost update speed?

edit: index

> db.invindex.stats() { "ns" : "spider.invindex", "count" : 595306, "size" : 1302386304, "avgobjsize" : 2187, "storagesize" : 1580150784, "numextents" : 19, "nindexes" : 1, "lastextentsize" : 415174656, "paddingfactor" : 1, "systemflags" : 1, "userflags" : 1, "totalindexsize" : 25648112, "indexsizes" : { "_id_" : 25648112 }, "ok" : 1 }

python mongodb pymongo mongodb-query

Comments

Post a Comment

Popular posts from this blog

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* -

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* - i've inherited doc publishing process takes docbook , builds html , pdf output, using apache fop both. project had been started, never completed. lots of tweaking, i've able of doc sets build (20+), except handful, , fail severe code 1 of these 2 conditions: "fo:flow" missing kid elements. required content model: marker* "fo:block" not valid kid of "fo:root" the xml output comes out of docbook , docbook-fop checks out well-formed. the fop error in 1 book, "endecagloss", example, points end of string: <fo:flow flow-name="xsl-region-body" start-indent="4pc" end-indent="0pt"/> here’s log output in case: [java] severe: javax.xml.transform.transformerexception: file:/scratch/publishing/hudson/jobs/build-main-endeca-documentation/workspace/serverdoc/w...
Read more

mediawiki - How do I insert tables inside infoboxes on Wikia pages? -

mediawiki - How do I insert tables inside infoboxes on Wikia pages? - i want insert tables in infoboxes. example, have proteinbox i've developed using infobox template (http://health-and-medicine.wikia.com/wiki/template:proteinbox), , fasta field big fit comfortably in proteinbox. hence collapsible table within box. this tried: {{infobox | row 1 = {| class = "mw-collapsible mw-collapsed wikitable" |- | <!-- table content --> |} }} your problem is, putting nested table within template. pipe characters ( | ) in table syntax collides pipe usage in templates. the commonly used hack around create template called template:! , containing pipe character, , utilize when need set tables, parser functions or other stuff using pipe characters, within templates. table (with every | replaced {{!}} ): {{{!}} {{!}}- {{!}} a1 {{!}} b1 {{!}}- {{!}} a2 {{!}} b2 {{!}}} ...the equivalent of {| |- | a1 | b1 |- | a2 | b2 |} furthermore, hav...
Read more

Local Service User Logged into Windows -

Local Service User Logged into Windows - i created windows service tracks app server user connected through load balancer. it retrieves user logged windows on machine, cross references active directory (ad) business relationship homecoming first name, lastly name , e-mail (for notification through website created project) there 1 scheme on network users remote system, vista system, reason returning values "local service" logged machine, it's weird , security guys impossible "local service" business relationship logged windows. here how user logged windows managementscope ms = new managementscope("\\\\.\\root\\cimv2"); objectquery query = new objectquery("select * win32_computersystem"); managementobjectsearcher searcher = new managementobjectsearcher(ms, query); foreach (managementobject mo in searcher.get()) { _username = mo["username"].tostring(); } // remove domain part username string[] usernameparts = _...
Read more