Breaking News: Grepper is joining You.com. Read the official announcement!

Turn off Spark logs in Spark-Submit

Cautious Chinchilla answered on November 3, 2024 Popularity 1/10 Helpfulness 1/10

answer Turn off Spark logs in Spark-Submit

Turn off Spark logs in Spark-Submit

Comment

spark-submit \
    --master yarn \
    --deploy-mode client \
    --driver-java-options "-Dlog4j.configuration=file:/absolute/path/to/your/log4j.properties" \
    --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \
    --files "/absolute/path/to/your/log4j.properties" \
    --class com.github.atais.Main \
    "SparkApp.jar"


#Also create log4j.properties file and add the following
log4j.rootLogger=ERROR, console

# set the log level for these components
log4j.logger.com.test=DEBUG
log4j.logger.org=ERROR
log4j.logger.org.apache.spark=ERROR
log4j.logger.org.spark-project=ERROR
log4j.logger.org.apache.hadoop=ERROR
log4j.logger.io.netty=ERROR
log4j.logger.org.apache.zookeeper=ERROR

# add a ConsoleAppender to the logger stdout to write to the console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
# use a simple message format
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

Popularity 1/10 Helpfulness 1/10 Language shell

Source: stackoverflow.com

Tags: shell spark-submit turn

Link to this answer
Share Copy Link

Contributed on Nov 03 2024

Cautious Chinchilla

0 Answers Avg Quality 2/10

Turn off Spark logs in Spark-Submit

Contents

More Related Answers

Turn off Spark logs in Spark-Submit

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.