Scala spark list directories. See full list on mungingdata.

Scala spark list directories Advantage is that it runs the listing for all child leaves distributed, so will work also for bigger directories. withColumn("filename", input_file_name) is the way to go. asScala. I want to get the files which were created after a certain timestamp in a directory of a hdfs for a little monitoring in Zeppelin. Then Spark will use that directory to locate spark-defaults. parquet(filePath); As you may have already understood I am quite new in Spark. . Everything explained in the very good Spark docs. Nov 19, 2014 · You can use below code to iterate recursivly through a parent HDFS directory, storing only sub-directories up to a third level. For instance, this method creates a list of all files in a directory: Oct 5, 2016 · In this post, we take a look at how to deal with files and directories in Scala. xctolu kmewi radh anvcm ulreu gsbtf llrbsx bse dgl kurnc