MongoDB findAndModify. Is it really atomic? Help writing a closed update solution -

MongoDB findAndModify. Is it really atomic? Help writing a closed update solution -

i have event documents, consisting of embedded snapshots.

i want add together snapshot event if:

that event started within 5 mins of snapshot a the latest snapshot on event not greater min before snapshot a.

otherwise.... create new event.

here findandupdate query might create more sense:

event.findandmodify( query: { start_timestamp: { $gte: newsnapshot.timestamp - 5min }, last_snapshot_timestamp: { $gte: newsnapshot.timestamp - 1min } }, update: { snapshots[newsnapshot.timestamp]: newsnapshot, $max: { last_snapshot_timestamp: newsnapshot.timestamp }, $min: { start_timestamp: newsnapshot.timestamp } }, upsert: true, $setoninsert: { our new event fields } } )

edit: unfortunately, cannot create unique index on start_timestamp. snapshots come in different timestamps, , want grouping them event. i.e snapshot comes in @ 12:00:00, , snapshot b comes in @ 12:00:59. should in same event, written db @ different times, because workers writing them acting concurrently. snapshot comes in, @ 12:00:30, should written same event 2 above. snapshot @ 12:02:00 should written new event.

my question is.... work correctly in concurrent environment. findandupdate atomic? possible might create 2 events, when should have created one, , added snapshot it?

edit: above approach not guaranteed not create 2 events, @chainh kindly pointed out.

so have tried new locking based approach - think work?

var acquirelock = function() { var query = { "locked": false} var update = { $set: { "locked": true } } homecoming lock.findandmodify({ query: query, update: update, upsert: true }) }; var releaselock = function() { var query = { "locked": true } var update = { $set: { "locked": false } } homecoming lock.findandmodify({ query: query, update: update }) }; var insertsnapshot = function(newsnapshot, upsert) { event.findandmodify( query: { start_timestamp: { $gte: newsnapshot.timestamp - 5min }, last_snapshot_timestamp: { $gte: newsnapshot.timestamp - 1min } }, update: { snapshots[newsnapshot.timestamp]: newsnapshot, $max: { last_snapshot_timestamp: newsnapshot.timestamp }, $min: { start_timestamp: newsnapshot.timestamp } }, upsert: upsert, $setoninsert: { our new event fields } } ) }; var safelyinsertevent = function(snapshot) { homecoming insertsnapshot(snapshot, false) .then(function(modifyres) { if (!modifyres.succeeded) { homecoming acquirelock() } }) .then(function(lockres) { if (lockres.succeeded) { homecoming insertsnapshot(snapshot, true) } else { throw new acquiringlockerror("didn't acquire lock. seek again") } }) .then(function() { homecoming releaselock() }) .catch(acquiringlockerror, function(err) { homecoming safelyinsertevent(snapshot) }) };

the lock document contain single field (locked). above code tries find existing event , update it. if works, great, can bail out. if didn't update, know don't have existing event stick snapshot in. acquire lock atomically, , if succeeds, can safely upsert new event. if acquiring lock fails, seek whole process again, , time have existing event stick in.

according codes:

event.findandmodify( query: { start_timestamp: { $gte: newsnapshot.timestamp - 5min }, last_snapshot_timestamp: { $gte: newsnapshot.timestamp - 1min } }, update: { snapshots[newsnapshot.timestamp]: newsnapshot, $max: { last_snapshot_timestamp: newsnapshot.timestamp }, $min: { start_timestamp: newsnapshot.timestamp } }, upsert: true, $setoninsert: { our new event fields } } )

when succeed insert first event document database, fields of event document has next relationship: start_timestamp == last_snapshot_timestamp

after subsequent updates, relationship turns to: start_timestamp < last_snapshot_timestamp < last_snapshot_timestamp + 1min < start_timestamp + 5min or start_timestamp < last_snapshot_timestamp < start_timestamp + 5min < last_snapshot_timestamp + 1min

so, if new snapshot wants insert event document continuously, must conform: newsnapshot.timestamp < math.min(last_snapshot_timestamp + 1, start_timestamp + 5)

suppose there 2 event documents in database on time: event1 (start_timestamp1, last_snapshot_timestamp1), event2 (start_timestamp2, last_snapshot_timestamp2) generally, start_timestamp2 > last_snapshot_timestamp1

now, if there new snapshot comes, , timestamp less start_timestamp1 (just suppose it's possible latency or forging), snapshot can inserted either event document. so, uncertainty whether need status added query part create sure distance between last_snapshot_timestamp , start_timestamp less value (e.g. 5min)? example, alter query

query: { start_timestamp: { $gte: newsnapshot.timestamp - 5min }, last_snapshot_timestamp: { $gte: newsnapshot.timestamp - 1min , $lte : newsnapshot.timestamp + 5} }

ok, let's continue... if seek solve question, still seek build unique index on field start_timestamp. according manual of mongodb, utilize findandmodify or update able finish work atomically. headache how should handle when duplicate value occurs because newsnapshot.timestamp out of command , perchance modify start_timestamp operator $min.

the approaches are:

several threads create(upsert) new event document because no documents can satisfy query condition; one thread succeed create new event document newsnapshot.timestamp value, others fail constraints of unique index on field start_timestamp; other threads retry (now update instead of upsert) , succeed update (use existed event document); if update(not upsert) causes modify start_timestamp $min operator , coincidentally newsnapshot.tiemstamp equal value of start_timestamp in existed event document, update fail constraints of unique index. can message, , know event document has existed, start_timestamp value equal newsnapshot.timestamp. now, can insert newsnapshot event document because conform condition.

as doesn't need homecoming event document, utilize update instead of findandmodify since both atomic operation , update has simpler writing in case. utilize simple javascript (run on mongo shell) express steps (i'm not familiar code syntax used. :d ), , think can understand easily.

var gap5 = 5 * 60 * 1000; // suppose, should alter accordingly if value not true. var gap1 = 1 * 60 * 1000; var initialfields = {}; // our new event fields function insertsnapshotifstarttimestampnotexisted() { var query = { start_timestamp: { $gte: newsnapshot.timestamp - gap5 }, last_snapshot_timestamp: { $gte: newsnapshot.timestamp - gap1 } }; var update = { $push : {snapshots: newsnapshot}, // suppose snapshots array $max: { last_snapshot_timestamp: newsnapshot.timestamp }, $min: { start_timestamp: newsnapshot.timestamp }, $setoninsert : initialfields }, var result = db.event.update(query, update, {upsert : true}); if (result.nupserted == 0 && result.nmodified == 0) { insertsnapshotifstarttimestampexisted(); // event document existed start_timestamp } } function insertsnapshotifstarttimestampexisted() { var query = { start_timestamp: newsnapshot.timestamp, }; var update = { $push : {snapshots: newsnapshot} }, var result = db.event.update(query, update, {upsert : false}); if (result.nmodified == 0) { insertsnapshotifstarttimestampnotexisted(); // if start_timestamp gets modified; it's possible. } } // entry db.event.ensureindex({start_timestamp:1},{unique:true}); insertsnapshotifstarttimestampnotexisted();

mongodb