Kafka with spark streaming integration error
up vote
0
down vote
favorite
I'm not able to run Kafka with spark-streaming. Following are the steps I've taken till now:
Downloaded the
jar
file "spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar" and moved it to/home/ec2-user/spark-2.0.0-bin-hadoop2.7/jars
Added this line to
/home/ec2-user/spark-2.0.0-bin-hadoop2.7/conf/spark-defaults.conf.template
->spark.jars.packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10:2.2.0
Kafka Version: kafka_2.10-0.10.2.2
Jar file version: spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar
Python Code:
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10-2.2.0 pyspark-shell'
kvs = KafkaUtils.createDirectStream(ssc, ["divolte-data"], {"metadata.broker.list": "localhost:9092"})
But I'm still getting the following error:
Py4JJavaError: An error occurred while calling o39.createDirectStreamWithoutMessageHandler.
: java.lang.NoClassDefFoundError: Could not initialize class kafka.consumer.FetchRequestAndResponseStatsRegistry$
at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:39)
at org.apache.spark.streaming.kafka.KafkaCluster.connect(KafkaCluster.scala:59)
What am I doing wrong?
java apache-spark pyspark apache-kafka spark-streaming
add a comment |
up vote
0
down vote
favorite
I'm not able to run Kafka with spark-streaming. Following are the steps I've taken till now:
Downloaded the
jar
file "spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar" and moved it to/home/ec2-user/spark-2.0.0-bin-hadoop2.7/jars
Added this line to
/home/ec2-user/spark-2.0.0-bin-hadoop2.7/conf/spark-defaults.conf.template
->spark.jars.packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10:2.2.0
Kafka Version: kafka_2.10-0.10.2.2
Jar file version: spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar
Python Code:
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10-2.2.0 pyspark-shell'
kvs = KafkaUtils.createDirectStream(ssc, ["divolte-data"], {"metadata.broker.list": "localhost:9092"})
But I'm still getting the following error:
Py4JJavaError: An error occurred while calling o39.createDirectStreamWithoutMessageHandler.
: java.lang.NoClassDefFoundError: Could not initialize class kafka.consumer.FetchRequestAndResponseStatsRegistry$
at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:39)
at org.apache.spark.streaming.kafka.KafkaCluster.connect(KafkaCluster.scala:59)
What am I doing wrong?
java apache-spark pyspark apache-kafka spark-streaming
How did you configure pom? Are you using ` metrics-core-2.2.0.jar?
spark-shell --.jars metrics-core-2.2.0.jar`
– karma4917
Nov 8 at 18:39
You're usingspark-2.0.0
, but your jars are for2.2.0
... Those versions should be the same
– cricket_007
Nov 9 at 15:18
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm not able to run Kafka with spark-streaming. Following are the steps I've taken till now:
Downloaded the
jar
file "spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar" and moved it to/home/ec2-user/spark-2.0.0-bin-hadoop2.7/jars
Added this line to
/home/ec2-user/spark-2.0.0-bin-hadoop2.7/conf/spark-defaults.conf.template
->spark.jars.packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10:2.2.0
Kafka Version: kafka_2.10-0.10.2.2
Jar file version: spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar
Python Code:
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10-2.2.0 pyspark-shell'
kvs = KafkaUtils.createDirectStream(ssc, ["divolte-data"], {"metadata.broker.list": "localhost:9092"})
But I'm still getting the following error:
Py4JJavaError: An error occurred while calling o39.createDirectStreamWithoutMessageHandler.
: java.lang.NoClassDefFoundError: Could not initialize class kafka.consumer.FetchRequestAndResponseStatsRegistry$
at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:39)
at org.apache.spark.streaming.kafka.KafkaCluster.connect(KafkaCluster.scala:59)
What am I doing wrong?
java apache-spark pyspark apache-kafka spark-streaming
I'm not able to run Kafka with spark-streaming. Following are the steps I've taken till now:
Downloaded the
jar
file "spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar" and moved it to/home/ec2-user/spark-2.0.0-bin-hadoop2.7/jars
Added this line to
/home/ec2-user/spark-2.0.0-bin-hadoop2.7/conf/spark-defaults.conf.template
->spark.jars.packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10:2.2.0
Kafka Version: kafka_2.10-0.10.2.2
Jar file version: spark-streaming-kafka-0-8-assembly_2.10-2.2.0.jar
Python Code:
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-streaming-kafka-0-8-assembly_2.10-2.2.0 pyspark-shell'
kvs = KafkaUtils.createDirectStream(ssc, ["divolte-data"], {"metadata.broker.list": "localhost:9092"})
But I'm still getting the following error:
Py4JJavaError: An error occurred while calling o39.createDirectStreamWithoutMessageHandler.
: java.lang.NoClassDefFoundError: Could not initialize class kafka.consumer.FetchRequestAndResponseStatsRegistry$
at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:39)
at org.apache.spark.streaming.kafka.KafkaCluster.connect(KafkaCluster.scala:59)
What am I doing wrong?
java apache-spark pyspark apache-kafka spark-streaming
java apache-spark pyspark apache-kafka spark-streaming
asked Nov 8 at 6:47
Jaskaran Singh Puri
187316
187316
How did you configure pom? Are you using ` metrics-core-2.2.0.jar?
spark-shell --.jars metrics-core-2.2.0.jar`
– karma4917
Nov 8 at 18:39
You're usingspark-2.0.0
, but your jars are for2.2.0
... Those versions should be the same
– cricket_007
Nov 9 at 15:18
add a comment |
How did you configure pom? Are you using ` metrics-core-2.2.0.jar?
spark-shell --.jars metrics-core-2.2.0.jar`
– karma4917
Nov 8 at 18:39
You're usingspark-2.0.0
, but your jars are for2.2.0
... Those versions should be the same
– cricket_007
Nov 9 at 15:18
How did you configure pom? Are you using ` metrics-core-2.2.0.jar
?
spark-shell --.jars metrics-core-2.2.0.jar`– karma4917
Nov 8 at 18:39
How did you configure pom? Are you using ` metrics-core-2.2.0.jar
?
spark-shell --.jars metrics-core-2.2.0.jar`– karma4917
Nov 8 at 18:39
You're using
spark-2.0.0
, but your jars are for 2.2.0
... Those versions should be the same– cricket_007
Nov 9 at 15:18
You're using
spark-2.0.0
, but your jars are for 2.2.0
... Those versions should be the same– cricket_007
Nov 9 at 15:18
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
spark-defaults.conf.template
is only a template, and not read by Spark, therefore your JARs will not be loaded. You must copy/rename this file to remove the template suffix
You'll also need to download Spark 2.2 if you want to use those specific JAR files.
And make sure that your Spark version uses Scala 2.10 if that's the Kafka package you want to use. Otherwise, use 2.11 version
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
spark-defaults.conf.template
is only a template, and not read by Spark, therefore your JARs will not be loaded. You must copy/rename this file to remove the template suffix
You'll also need to download Spark 2.2 if you want to use those specific JAR files.
And make sure that your Spark version uses Scala 2.10 if that's the Kafka package you want to use. Otherwise, use 2.11 version
add a comment |
up vote
0
down vote
spark-defaults.conf.template
is only a template, and not read by Spark, therefore your JARs will not be loaded. You must copy/rename this file to remove the template suffix
You'll also need to download Spark 2.2 if you want to use those specific JAR files.
And make sure that your Spark version uses Scala 2.10 if that's the Kafka package you want to use. Otherwise, use 2.11 version
add a comment |
up vote
0
down vote
up vote
0
down vote
spark-defaults.conf.template
is only a template, and not read by Spark, therefore your JARs will not be loaded. You must copy/rename this file to remove the template suffix
You'll also need to download Spark 2.2 if you want to use those specific JAR files.
And make sure that your Spark version uses Scala 2.10 if that's the Kafka package you want to use. Otherwise, use 2.11 version
spark-defaults.conf.template
is only a template, and not read by Spark, therefore your JARs will not be loaded. You must copy/rename this file to remove the template suffix
You'll also need to download Spark 2.2 if you want to use those specific JAR files.
And make sure that your Spark version uses Scala 2.10 if that's the Kafka package you want to use. Otherwise, use 2.11 version
answered Nov 9 at 15:20
cricket_007
77.7k1142108
77.7k1142108
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53202704%2fkafka-with-spark-streaming-integration-error%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How did you configure pom? Are you using ` metrics-core-2.2.0.jar
?
spark-shell --.jars metrics-core-2.2.0.jar`– karma4917
Nov 8 at 18:39
You're using
spark-2.0.0
, but your jars are for2.2.0
... Those versions should be the same– cricket_007
Nov 9 at 15:18