使用pyspark读取hive中的数据，测试代码：
vi test.py
#!-*- coding:utf-8 -*-
from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext
conf = (SparkConf().setMaster("yarn").setAppName("My app").set("spark.executor.memory", "1g"))
sc = SparkContext(conf = conf)
sqlContext = HiveContext(sc)
my_dataframe = sqlContext.sql("Select * from database.table limit 10")
my_dataframe.show()
sc.stop()报错：
python pyspark_hive.py
Traceback (most recent call last):
File “pyspark_hive.py”, line 2, in <module>
from pyspark.sql import HiveContext,SparkSession
ModuleNotFoundError: No module named ‘pyspark’
环境变量设置：
vi ~/.profile
export SPARK_HOME=/usr/lib/spark-current
export PYTHONPATH=$SPARK_HOME/python/:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH参考：
https://www.cnblogs.com/tong775131501/p/7582258.html
正文完