Skip to main content

Spark SQL Settings

AtScale supports the following settings for configuring SparkSQL connections.

sparkSql.hiveDialect​

  • Default: 0.13
  • Restart: No

The version of Hive on which SparkSQL is built.

sparkSql.shufflePartitions.system_query_role.enabled​

  • Default: false
  • Restart: No

If enabled it will run a SET spark.sql.shuffle.partitions before each query that runs on the system role.

sparkSql.shufflePartitions.canary_query_role.enabled​

  • Default: false
  • Restart: No

If enabled it will run a SET spark.sql.shuffle.partitions before each query that runs on the canary role.

sparkSql.shufflePartitions.system_query_role.partitions​

  • Default: 150
  • Restart: No

The number of shuffle partitions to set for queries that run on the system role.

sparkSql.shufflePartitions.small_user_query_role.enabled​

  • Default: false
  • Restart: No

If enabled it will run a SET spark.sql.shuffle.partitions before each query that runs on the small interactive role.

sparkSql.settings.adaptiveExecution.enabled​

  • Default: true
  • Restart: No

If enabled it will run a SET spark.sql.adaptive.enabled for each connection.

sparkSql.settings.crossJoin.enabled​

  • Default: true
  • Restart: No

If enabled it will run a SET spark.sql.crossJoin.enabled for each connection.

sparkSql.shufflePartitions.small_user_query_role.partitions​

  • Default: 10
  • Restart: No

The number of shuffle partitions to set for queries that run on the small interactive role.

sparkSql.shufflePartitions.canary_query_role.partitions​

  • Default: 10
  • Restart: No

The number of shuffle partitions to set for queries that run on the canary role.

sparkSql.shufflePartitions.large_user_query_role.partitions​

  • Default: 150
  • Restart: No

The number of shuffle partitions to set for queries that run on the large interactive role.

sparkSql.shufflePartitions.large_user_query_role.enabled​

  • Default: false
  • Restart: No

If enabled it will run a SET spark.sql.shuffle.partitions before each query that runs on the large interactive role.