我正在尝试使用Avro文件将数据大容量加载到Snowflake中。我可以创建文件并将其放入Snowflake阶段,但当我运行命令时
copy into MyTable from '@%"MyTable"/LinkTypes.avro' file_format = (type = avro) match_by_column_name = case_insensitive;
我错了:
100084(22P02):分析AVRO时出错:不是AVRO文件(魔术不好)
在十六进制编辑器中查看这个文件,我没有注意到它有什么特别糟糕的地方(
Microsoft.Avro.Core
来自NuGet),并且它加载得很好
this random online Avro viewer
我在谷歌上搜索了一下。但不管出于什么原因,雪花都看不懂。
The file can be found here.
我的Avro数据生成例程如下。有人知道它出了什么问题以及如何修复吗?
public void Write(string name, IDataReader data, DataDictionary dict)
{
using var stream = OpenFileStream(name); // returns a FileStream; assume this works
var schema = BuildAvroSchema(dict.GetSchema(name)); // returns a JSON string describing the schema; assume this works
using var writer = AvroContainer.CreateGenericWriter(schema, stream, Codec.Deflate);
var serializer = AvroSerializer.CreateGeneric(schema);
WriteReader(reader, writer, stream, serializer.WriterSchema);
}
private static void WriteReader(IDataReader reader, IAvroWriter<object> writer, Stream stream, TypeSchema schema)
{
using var sw = new SequentialWriter<object>(writer, 1024);
var rec = new AvroRecord(schema);
var len = reader.FieldCount;
var buffer = new object[len];
var names = Enumerable.Range(0, len).Select(i => reader.GetName(i)).ToArray();
while (reader.Read()) {
reader.GetValues(buffer);
for (int i = 0; i < len; ++i) {
var value = buffer[i];
if (value is DateTime dt) {
value = dt.Ticks;
}
rec[names[i]] = value == DBNull.Value ? null : value;
}
sw.Write(rec);
}
}