datastax enterprise - DSE Hadoop intermittent timeout errors during read -
datastax enterprise - DSE Hadoop intermittent timeout errors during read -
i having weird error has started happening few weeks ago. had replace several analytics nodes , none of hadoop jobs invoked hive able finish. crash on different stages similar error:
com.datastax.driver.core.exceptions.nohostavailableexception: host(s) tried query failed (tried: ip-x-x-x-x.ec2.internal/x.x.x.x:9042 (com.datastax.driver.core.exceptions.driverexception: timeout during read)) @ com.datastax.driver.core.exceptions.nohostavailableexception.copy(nohostavailableexception.java:65) @ com.datastax.driver.core.defaultresultsetfuture.extractcausefromexecutionexception(defaultresultsetfuture.java:256) @ com.datastax.driver.core.arraybackedresultset$multipage.preparenextrow(arraybackedresultset.java:259) @ com.datastax.driver.core.arraybackedresultset$multipage.isexhausted(arraybackedresultset.java:222) @ com.datastax.driver.core.arraybackedresultset$1.hasnext(arraybackedresultset.java:115) @ org.apache.cassandra.hadoop.cql3.cqlrecordreader$rowiterator.computenext(cqlrecordreader.java:239) @ org.apache.cassandra.hadoop.cql3.cqlrecordreader$rowiterator.computenext(cqlrecordreader.java:218) @ com.google.common.collect.abstractiterator.trytocomputenext(abstractiterator.java:143) @ com.google.common.collect.abstractiterator.hasnext(abstractiterator.java:138) @ org.apache.cassandra.hadoop.cql3.cqlrecordreader.getprogress(cqlrecordreader.java:152) @ org.apache.hadoop.hive.cassandra.cql3.input.cqlhiverecordreader.getprogress(cqlhiverecordreader.java:62) @ org.apache.hadoop.hive.ql.io.hiverecordreader.getprogress(hiverecordreader.java:71) @ org.apache.hadoop.mapred.maptask$trackedrecordreader.getprogress(maptask.java:260) @ org.apache.hadoop.mapred.maptask$trackedrecordreader.movetonext(maptask.java:233) @ org.apache.hadoop.mapred.maptask$trackedrecordreader.next(maptask.java:216) @ org.apache.hadoop.mapred.maprunner.run(maprunner.java:48) @ org.apache.hadoop.mapred.maptask.runoldmapper(maptask.java:436) @ org.apache.hadoop.mapred.maptask.run(maptask.java:372) @ org.apache.hadoop.mapred.child$4.run(child.java:266) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:415) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1121) @ org.apache.hadoop.mapred.child.main(child.java:260) caused by: com.datastax.driver.core.exceptions.nohostavailableexception: host(s) tried query failed (tried: ip-x-x-x-x.ec2.internal/x.x.x.x:9042 (com.datastax.driver.core.exceptions.driverexception: timeout during read)) @ com.datastax.driver.core.requesthandler.sendrequest(requesthandler.java:103) @ com.datastax.driver.core.requesthandler$1.run(requesthandler.java:175) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:744)
i turned debug logging, still not find happening around time.
thanks!
actually, problem in huge amount of info written application 1 of map columns. happen co coincide cluster update. hive jobs hanging , there misleading error message. trial , error able narrow downwards issue map column , delete offending data.
hadoop datastax-enterprise datastax
Comments
Post a Comment