apache spark - How to convert Scala RDD to Map -



apache spark - How to convert Scala RDD to Map -

i have rdd (array of string) org.apache.spark.rdd.rdd[string] = mappedrdd[18] , convert map uniqueids did 'val vertexmap = vertices.zipwithuniqueid' gave me rdd of type 'org.apache.spark.rdd.rdd[(string, long)]' want 'map[string,long]' . how can convert 'org.apache.spark.rdd.rdd[(string, long)] map[string,long]' ?

thanks

there's built-in collectasmap function in pairrddfunctions deliver map of pair values in rdd.

val vertexmap = vertices.zipwithuniqueid.collectasmap

it's of import remember rdd distributed info structure. can visualize 'pieces' of info spread on cluster. when collect, forcefulness pieces go driver , able that, need fit in memory of driver.

from comments, looks in case, need deal big dataset. making map out of not going work won't fit on driver's memory; causing oom exceptions if try.

you need maintain dataset rdd. if creating map in order lookup elements, utilize lookup on pairrdd instead, this:

import org.apache.spark.sparkcontext._ // import implicits conversions back upwards pairrddfunctions val vertexmap = vertices.zipwithuniqueid val vertixyid = vertexmap.lookup("vertexy")

scala apache-spark

Comments

Post a Comment

Popular posts from this blog

php - How to pass multiple values from url -

php - How to pass multiple values from url - i'm trying pass textbox , 1 select result value 1 page page.using url.but it's display error message. "parse error: syntax error, unexpected '' (t_encapsed_and_whitespace), expecting identifier (t_string) or variable (t_variable) or number (t_num_string) in c:.." while($value= mysqli_fetch_assoc($result)){ $apid=$value['apid']; echo("<tr> <td>". $value['apid'] ."</td> <td>".$value['email']."</td> <td>".$value['comments']."</td> <td>".'<input type="text" name="payment"/>'."</td> <td>"."<input type='text' name='apid' value='".$apid."' style='visibility:hidden' size='2'>"."</a>"."<a hre...
Read more

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* -

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* - i've inherited doc publishing process takes docbook , builds html , pdf output, using apache fop both. project had been started, never completed. lots of tweaking, i've able of doc sets build (20+), except handful, , fail severe code 1 of these 2 conditions: "fo:flow" missing kid elements. required content model: marker* "fo:block" not valid kid of "fo:root" the xml output comes out of docbook , docbook-fop checks out well-formed. the fop error in 1 book, "endecagloss", example, points end of string: <fo:flow flow-name="xsl-region-body" start-indent="4pc" end-indent="0pt"/> here’s log output in case: [java] severe: javax.xml.transform.transformerexception: file:/scratch/publishing/hudson/jobs/build-main-endeca-documentation/workspace/serverdoc/w...
Read more

database - php search bar when I press submit with nothing in the search bar it shows all the data -

database - php search bar when I press submit with nothing in the search bar it shows all the data - the search fusion works correctly if press submit nil in search bar shows data. how message show saying nil has been entered search bar ?.i new php. <?php $mysql_host="host"; $mysql_database="database"; $mysql_user="username"; $mysql_password="password"; $dbconnect=@mysql_connect($mysql_host, $mysql_user, $mysql_password); // trys connect database if (!$dbconnect) { exit("an error has occurred - not connect database."); // if couldn't connect, allow user know } if(!mysql_select_db($mysql_database)) { exit("an error has occurred - not select database."); //if couldn't select db, allow user know } $output = ''; //collect if(isset($_post['search'])...
Read more