java - How to print first and last line of file in hadoop? -



java - How to print first and last line of file in hadoop? -

what best way print first line , lastly line of input file using hadoop map cut down ?

for illustration if have file of 10 gb , typical block size 128 mb approximately 80 mappers invoked keeping default configuration means not manipulating split size

so 80 mappers invoked how differentiate how framework has assigned split size means starting split size offset or number mapper.

so can't set logic in map function blindly way applied other mappers .

one solution can think of using 1 mapper keeping block size of file size way can set functionality in map function way won't able create utilize of parallel computing .

any effective way of doing ?

can seek "hadoop fs" commands separately store first , lastly line , run map cut down jobs on it. hadoop has specific tail command straight gives lastly n lines in file.

this tried:

file size: 2.2mb

first line: getting first straight forward, cat , take head -n1 hadoop fs -cat $file | head -n1 time taken: 4s

last line: there 2 ways this, 1 cat , tail. since file size if big long.

hadoop fs -cat $file | tail -n1

time taken: 39 seconds

but luckily there tail command comes rescue here. can hadoo fs -tail on file , time taken same head command. per documentation : displays lastly kilobyte of file stdout. -f alternative can used in unix

hadoop fs -tail | $file | tail -n1

time taken: 4 seconds

you can seek on file , check time difference.

java hadoop mapreduce

Comments

Post a Comment

Popular posts from this blog

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* -

xslt - DocBook 5 to PDF transform failing with error: "fo:flow" is missing child elements. Required content model: marker* - i've inherited doc publishing process takes docbook , builds html , pdf output, using apache fop both. project had been started, never completed. lots of tweaking, i've able of doc sets build (20+), except handful, , fail severe code 1 of these 2 conditions: "fo:flow" missing kid elements. required content model: marker* "fo:block" not valid kid of "fo:root" the xml output comes out of docbook , docbook-fop checks out well-formed. the fop error in 1 book, "endecagloss", example, points end of string: <fo:flow flow-name="xsl-region-body" start-indent="4pc" end-indent="0pt"/> here’s log output in case: [java] severe: javax.xml.transform.transformerexception: file:/scratch/publishing/hudson/jobs/build-main-endeca-documentation/workspace/serverdoc/w...
Read more

mediawiki - How do I insert tables inside infoboxes on Wikia pages? -

mediawiki - How do I insert tables inside infoboxes on Wikia pages? - i want insert tables in infoboxes. example, have proteinbox i've developed using infobox template (http://health-and-medicine.wikia.com/wiki/template:proteinbox), , fasta field big fit comfortably in proteinbox. hence collapsible table within box. this tried: {{infobox | row 1 = {| class = "mw-collapsible mw-collapsed wikitable" |- | <!-- table content --> |} }} your problem is, putting nested table within template. pipe characters ( | ) in table syntax collides pipe usage in templates. the commonly used hack around create template called template:! , containing pipe character, , utilize when need set tables, parser functions or other stuff using pipe characters, within templates. table (with every | replaced {{!}} ): {{{!}} {{!}}- {{!}} a1 {{!}} b1 {{!}}- {{!}} a2 {{!}} b2 {{!}}} ...the equivalent of {| |- | a1 | b1 |- | a2 | b2 |} furthermore, hav...
Read more

Local Service User Logged into Windows -

Local Service User Logged into Windows - i created windows service tracks app server user connected through load balancer. it retrieves user logged windows on machine, cross references active directory (ad) business relationship homecoming first name, lastly name , e-mail (for notification through website created project) there 1 scheme on network users remote system, vista system, reason returning values "local service" logged machine, it's weird , security guys impossible "local service" business relationship logged windows. here how user logged windows managementscope ms = new managementscope("\\\\.\\root\\cimv2"); objectquery query = new objectquery("select * win32_computersystem"); managementobjectsearcher searcher = new managementobjectsearcher(ms, query); foreach (managementobject mo in searcher.get()) { _username = mo["username"].tostring(); } // remove domain part username string[] usernameparts = _...
Read more