Limitations and Supported Types
General limitations
- Spark Thrift Server and Livy Thrift Server are not supported.
- Spark UDFs are not supported when queries are executed in Hive.
- Hive UNION types are not supported.
DataFrame API limitations
When writing with the HWC DataFrame writer:
- Table statistics are not automatically generated for the target Hive table.
- You cannot read from and overwrite the same table in a single operation.
- In Spark 3, HWC does not auto-create tables for append/overwrite.
Direct reader limitations
Direct reader is optimized for speed but has functional limits:
- It does not enforce Ranger or HS2 authorization policies.
- It does not support writes or streaming inserts.
- It guarantees a consistent snapshot per table, not across multiple tables.
JDBC mode limitations
JDBC mode is best for small or moderate result sets. For large data extracts, use direct reader or secure access. In client mode, all results flow through the driver.
Supported types
The following Spark types are supported when reading or writing Hive tables:
| Spark Type | Hive Type |
|---|---|
| ByteType | TinyInt |
| ShortType | SmallInt |
| IntegerType | Integer |
| LongType | BigInt |
| FloatType | Float |
| DoubleType | Double |
| DecimalType | Decimal |
| StringType | String, Char, Varchar |
| BinaryType | Binary |
| BooleanType | Boolean |
| TimestampType | Timestamp |
| DateType | Date |
| ArrayType | Array |
| StructType | Struct |
Unsupported types:
- CalendarIntervalType (Hive interval)
- MapType (Hive map)
- Union types
- NullType