Caused by: java.lang.OutOfMemoryError: Notenough memory to build and broadcast the table to all work nodes. As a workarround, you can either disable broadcast...

同时在开发环境跑Spark作业的时候遇到了以下报错，让我帮忙看看有没有思路

Caused by: java.lang.OutOfMemoryError: Notenough memory to build and broadcast the table to all work nodes. As a workarround, you can either disable broadcast by setting spark.sql.autoBroadcaseJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value at ...

此时driver memory用的是32G, 同事说如果把memory增大到64G会有别的报错：

Caused by: org.apache.sark.SparkException: Cannot broadcast the table that is larger than 8GB: 33 GB...

查看作业代码，同事不小心将大表broadcast，修复改为小表broadcast，作业执行成功，问题解决。