Enable streaming / CDF streaming reads on column mapping enabled tables with fewer limitations

[jackierwzhang]

Description

Resolves #1357

As streaming uses the latest schema to read historical data batches and column mapping schema changes (e.g. rename/drop column) can cause latest schema to diverge, we decided to temporarily completely block streaming read on column mapping tables before.

As a close follow up in this PR, we think it is at least possible to enable the following use cases:

Read from a column mapping table without rename or drop column operations.
Upgrade to column mapping tables.
Existing compatible schema change operations such as ADD COLUMN.

How was this patch tested?

New unit tests.

Does this PR introduce any user-facing changes?

No

[scottsand-db]

Looks great! Thanks for doing this. Comments, code, and tests are all very clean and thorough. Left a few minor minor questions/comments.

[tnk-dev]

Hey Delta folks,

since the only way to allow renaming and dropping columns in delta without rewriting everything is to enable columnMapping, which collides with Change Data Feed resulting in
com.databricks.sql.transaction.tahoe.DeltaColumnMappingUnsupportedSchemaIncompatibleException: Change Data Feed (CDF) read is not supported on tables with column mapping schema changes (e.g. rename or drop)...
and can only be solved with spark.databricks.delta.changeDataFeed.unsafeBatchReadOnIncompatibleSchemaChanges.enabled, we as a company are kinda scared of enabling this option and are not sure at all about the ramifications of enabling unsafeBatchReadOnIncompatibleSchemaChanges. Plus we don't really like the 2 character partitioning convention in our Datalake (S3), that is replaced by the columnMapping option.

So my question is:
What is best practice here according to Delta regarding above mentioned pain points?

@dennyglee