How could I write the right entry point in Spark 2.0 program (Actually pyspark 2.0)? -

-

today, wanna try new features spark2.0, here program:

#coding:utf-8 pyspark.conf import sparkconf pyspark.sql import sparksession  spark = sparksession.builder.master("local").appname('test 2.0').config(conf=sparkconf()).getorcreate() df = spark.read.json("/users/lyj/programs/apache/spark2/examples/src/main/resources/people.json") df.show()

but errors follow:

traceback (most recent call last):   file "/users/lyj/programs/kiseliugit/mypysparkcodes/test/spark2.0.py", line 5, in <module> spark = sparksession.builder.master("local").appname('test 2.0').config(conf=sparkconf()).getorcreate()   file "/users/lyj/programs/apache/spark2/python/pyspark/conf.py", line 104, in __init__ sparkcontext._ensure_initialized()   file "/users/lyj/programs/apache/spark2/python/pyspark/context.py", line 243, in _ensure_initialized sparkcontext._gateway = gateway or launch_gateway()   file "/users/lyj/programs/apache/spark2/python/pyspark/java_gateway.py", line 116, in launch_gateway java_import(gateway.jvm, "org.apache.spark.sparkconf")   file "/library/python/2.7/site-packages/py4j/java_gateway.py", line 90, in java_import return_value = get_return_value(answer, gateway_client, none, none)   file "/library/python/2.7/site-packages/py4j/protocol.py", line 306, in get_return_value value = output_converter[type](answer[2:], gateway_client) keyerror: u'y'

what's wrong these few lines of codes? have problem java environment? plus, use ide pycharm developing.

try upgrade py4j, pip install py4j --upgrade

it's worked me.


Comments

Post a Comment

Popular posts from this blog

Lists in Python -

-
i noticed can multiply list scalar behavior weird if multiply: [2,3,4] * 3 i get: [2,3,4,2,3,4,2,3,4 ] understand results it's for? there other weird operations that? the main purpose of operand initialisation. example, if want initialize list 20 equal numbers can using loop: arr=[] in range(20): arr.append(3) an alternative way using operator: arr = [3] * 20 more weird , normal list operation on lists can find here http://www.discoversdk.com/knowledge-base/using-lists-in-python
Read more

android - can not access to progress bar in an other activity -

-
i have objects in layout video view,text view , on comes this: <?xml version="1.0" encoding="utf-8"?> <android.support.v7.widget.cardview android:id="@+id/cardview_mypost" android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_marginbottom="@dimen/activity_vertical_margin" app:cardcornerradius="@dimen/activity_vertical_margin" app:cardelevation="@dimen/activity_vertical_margin" xmlns:android="http://schemas.android.com/apk/res/android" xmlns:app="http://schemas.android.com/apk/res-auto"> <relativelayout xmlns:android="http://schemas.android.com/apk/res/android" android:layout_width="fill_parent" android:layout_height="wrap_content" android:background="@color/web_withe_color" android:animatelayoutchanges="@anim/anim_list" android:padding="8dp" > <!-- thu...
Read more

html - Not able to access next element of an array javascript -

-
i new js, trying access next element of array using onclick function no luck. var i, len; function quiz() { var quiz_questions = [ "who founder of facebook?", "who founder of google?" ]; len = quiz_questions.length; for(i = 0; < len; i++) { document.getelementbyid("demo").innerhtml = quiz_questions[i]; } } <button onclick="quiz();">click</button> <p id="demo"></p> can explain me did wrong here. your problem is, iterate through every question, every time click. remove loop , put i++ after setting question. maybe checking, if less array length before it, wouldnt bad idea. var i, len; = 0; function quiz() { var quiz_questions = [ "who founder of facebook?", "who founder of google?" ]; len = quiz_questions.length; document.gete...
Read more