Wsrtjtyk

I've tried the following in Jupyter in order to read in the CSV file in a table format.

pyspark --packages com.databricks:spark-csv_2.10:1.5.0

then I got the following error in the log, for more details about the log "i've listed separately in the next comment"

:::: WARNINGS

module not found: com.databricks#spark-csv_2.10;1.5.0

"I've checked spark-csv_2.10-1.5.0.jar", and "commons-csv-1.1.jar" are already exist

if i ignored the warning, i got this error "NameError: name 'sc' is not defined" when running the following

sqlContext = SQLContext(sc)

and I'm really stuck, thus any suggestion, please.
the target is to read in the CSV file as below

sqlContext = SQLContext(sc)

data = sqlContext.read.load('file:///path/file.csv', format='com.databricks.spark.csv', header='true',inferSchema='true')

Here is the Log:

pyspark --packages com.databricks:spark-csv_2.10:1.5.0

/home/cloudera/.local/lib/python3.5/site-packages/requests/init.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 3]) may cause slowdown.

warnings.warn(warning, RequestsDependencyWarning)

[I 10:32:29.300 NotebookApp] The port 8888 is already in use, trying another random port.

[I 10:32:29.311 NotebookApp] Serving notebooks from local directory: /home/cloudera/Downloads/coursera-master/big-data-4

[I 10:32:29.312 NotebookApp] 0 active kernels

[I 10:32:29.312 NotebookApp] The Jupyter Notebook is running at: http://localhost:8889/

[I 10:32:29.312 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

WARNING: content window passed to PrivateBrowsingUtils.isWindowPrivate. Use isContentWindowPrivate instead (but only for frame scripts).

pbu_isWindowPrivate@resource://gre/modules/PrivateBrowsingUtils.jsm:25:14

nsBrowserAccess.prototype.openURI@chrome://browser/content/browser.js:15192:21

NewNotebookWidget.prototype.new_notebook@http://localhost:8889/static/tree/js/main.min.js?v=cee9d5ded70fc8733bb888581c22f633:15194:17

.proxy/i@http://localhost:8889/static/tree/js/main.min.js?v=cee9d5ded70fc8733bb888581c22f633:4:5486

x.event.dispatch@http://localhost:8889/static/tree/js/main.min.js?v=cee9d5ded70fc8733bb888581c22f633:5:9954

x.event.add/y.handle@http://localhost:8889/static/tree/js/main.min.js?v=cee9d5ded70fc8733bb888581c22f633:5:6772

[I 10:32:35.674 NotebookApp] Creating new notebook in

[I 10:32:36.695 NotebookApp] Kernel started: 25ed0b47-e0f0-4191-b1bc-984679f2668c

Ivy Default Cache set to: /home/cloudera/.ivy2/cache

The jars for the packages stored in: /home/cloudera/.ivy2/jars

:: loading settings :: url = jar:file:/usr/lib/spark/lib/spark-assembly-1.6.0-cdh5.16.0-hadoop2.6.0-cdh5.16.0.jar!/org/apache/ivy/core/settings/ivysettings.xml

com.databricks#spark-csv_2.10 added as a dependency

:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0

confs: [default]

[W 10:32:47.059 NotebookApp] Timeout waiting for kernel_info reply from 25ed0b47-e0f0-4191-b1bc-984679f2668c

:: resolution report :: resolve 8250ms :: artifacts dl 0ms

:: modules in use:

---------------------------------------------------------------------

| | modules || artifacts |

| conf | number| search|dwnlded|evicted|| number|dwnlded|

---------------------------------------------------------------------

| default | 1 | 0 | 0 | 0 || 0 | 0 |

---------------------------------------------------------------------



:: problems summary ::

:::: WARNINGS

module not found: com.databricks#spark-csv_2.10;1.5.0



==== local-m2-cache: tried



  file:/home/cloudera/.m2/repository/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.pom



  -- artifact com.databricks#spark-csv_2.10;1.5.0!spark-csv_2.10.jar:



  file:/home/cloudera/.m2/repository/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.jar



==== local-ivy-cache: tried



  /home/cloudera/.ivy2/local/com.databricks/spark-csv_2.10/1.5.0/ivys/ivy.xml



==== central: tried



  https://repo1.maven.org/maven2/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.pom



  -- artifact com.databricks#spark-csv_2.10;1.5.0!spark-csv_2.10.jar:



  https://repo1.maven.org/maven2/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.jar



==== spark-packages: tried



  http://dl.bintray.com/spark-packages/maven/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.pom



  -- artifact com.databricks#spark-csv_2.10;1.5.0!spark-csv_2.10.jar:



  http://dl.bintray.com/spark-packages/maven/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.jar



    ::::::::::::::::::::::::::::::::::::::::::::::



    ::          UNRESOLVED DEPENDENCIES         ::



    ::::::::::::::::::::::::::::::::::::::::::::::



    :: com.databricks#spark-csv_2.10;1.5.0: not found



    ::::::::::::::::::::::::::::::::::::::::::::::

:::: ERRORS

Server access error at url https://repo1.maven.org/maven2/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)



Server access error at url https://repo1.maven.org/maven2/com/databricks/spark-csv_2.10/1.5.0/spark-csv_2.10-1.5.0.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: com.databricks#spark-csv_2.10;1.5.0: not found]

at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1067)

at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /usr/lib/spark/python/pyspark/shell.py:

I think you can use another way to read csv files in pyspark by:

spark.read.csv("yourPath", header=True)

and do not need to import others packages.