首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >SparkContext:初始化SparkContext时出错。java.net.UnknownHostException:主站

SparkContext:初始化SparkContext时出错。java.net.UnknownHostException:主站
EN

Stack Overflow用户
提问于 2016-08-24 11:53:25
回答 1查看 4K关注 0票数 2

我是新的Spark和Intellij下面是我的build.sbt文件:

代码语言:javascript
复制
name := "TestSpark"
version := "1.0"
scalaVersion := "2.11.8"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.11" % "1.6.2","org.apache.hadoop" % "hadoop-client" % "2.6.2")

我在src/main/scala-2.11/TestMain.scala中也有TestMain.scala

代码语言:javascript
复制
import org.apache.spark.{SparkContext,SparkConf}
/**
  * Created by tuannv5 on 24/08/2016.
  */
object TestMain {
  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local[*]").setAppName("Test Spark").set("spark.executor.memory","2g")
    val sc = new SparkContext(conf)
    val data = sc.parallelize(1 to 1000000).filter(_<10000)
    data.foreach(println)

  }
}

由于某些原因,当我在Intellij中运行应用程序时。我得到了错误:

代码语言:javascript
复制
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/08/24 10:29:28 INFO SparkContext: Running Spark version 1.6.2
16/08/24 10:29:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/08/24 10:29:29 ERROR SparkContext: Error initializing SparkContext.
java.net.UnknownHostException: master: master: unknown error
    at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
    at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:789)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:782)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:782)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.util.Utils$.localHostName(Utils.scala:839)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:420)
    at TestMain$.main(TestMain.scala:8)
    at TestMain.main(TestMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.net.UnknownHostException: master: unknown error
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
    ... 15 more
16/08/24 10:29:29 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.net.UnknownHostException: master: master: unknown error
    at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
    at org.apache.spark.util.Utils$.findLocalInetAddress(Utils.scala:789)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress$lzycompute(Utils.scala:782)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$localIpAddress(Utils.scala:782)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839)
    at org.apache.spark.util.Utils$$anonfun$localHostName$1.apply(Utils.scala:839)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.util.Utils$.localHostName(Utils.scala:839)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:420)
    at TestMain$.main(TestMain.scala:8)
    at TestMain.main(TestMain.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.net.UnknownHostException: master: unknown error
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
    at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
    ... 15 more

Process finished with exit code 1

有人能给我解释一下这个错误中提到的问题吗?这是因为我的依赖项没有正确安装,还是因为其他原因?

EN

回答 1

Stack Overflow用户

发布于 2017-01-18 08:20:10

您的主机名不正确。

键入主机名,并检查它在/etc/hosts中是否相同:

例如:

代码语言:javascript
复制
root@google:/home/makbul# hostname

谷歌在/etc/hosts上也是如此

代码语言:javascript
复制
127.0.1.1       google
票数 3
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/39114140

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档