代码之家 › 专栏 › 技术社区 › Woody1193 Nimmi Rashinika

正在将外部JSON负载反序列化为protobuf-Any

protocol-buffers go json

Woody1193 Nimmi Rashinika · 技术社区 · 2 年前

我有一个protobuf定义来处理API的分页结果:

message ArrayRespone {
    int32 count = 1;
    string next_url = 2;
    string request_id = 3;
    repeated google.protobuf.Any results = 4;
    string status = 5;
}

这里的目标是反序列化来自这个API的分页响应,然后将每个页面的结果提取到适当类型的切片中。我在Go中编写了这样的代码:

func getData[T ~proto.Message](data []byte) ([]T, error) {

    var resp *ArrayRespone
    if err := json.Unmarshal(data, &resp); err != nil {
        return nil, err
    }
    
    var items []T
    for _, result := range resp.Results {
        var item T
        if err := result.UnmarshalTo(item); err != nil {
            return nil, err
        }

        items = append(items, item)
    }

    return items, nil
}

我遇到的问题是,在测试此代码时,我遇到了以下错误:

proto:不匹配的消息类型:得到“X”,想要“”

由此,我可以理解Protobuf没有必要的信息来确定它使用的是哪种类型。查看的定义 Any ,我可以看到它有一个 TypeUrl 字段和 Value 领域类型URL似乎是空的,但不应该是。所以,我的想法是,如果我将其设置为 X ,错误会消失,但这也不起作用,因为 价值 田地还是空的;我的JSON数据被忽略了。

如何使此代码正常工作?

0 回复 | 直到 2 年前

Woody1193 Nimmi Rashinika 2 年前

我找到了两个解决这个问题的潜在方案,但它们都涉及的自定义实现 UnmarshalJSON 。首先,我尝试修改我的原型定义 results 属于类型 bytes ,但JSON反序列化失败,因为源数据不是字符串或任何可以反序列化为 []byte 直接地所以,我不得不自己打滚:

使用Struct

使用 google.protobuf.Struct 类型,我修改了 ArrayResponse 看起来像这样:

message ArrayRespone {
    int32 count = 1;
    string next_url = 2;
    string request_id = 3;
    repeated google.protobuf.Struct results = 4;
    string status = 5;
}

然后编写了的自定义实现 取消聚合JSON 它是这样工作的:

// UnmarshalJSON converts JSON data into a Providers.Polygon.ArrayResponse
func (resp *ArrayRespone) UnmarshalJSON(data []byte) error {

    // First, deserialize the JSON into a mapping between key fields and values
    // If this fails then return an error
    var mapped map[string]interface{}
    if err := json.Unmarshal(data, &mapped); err != nil {
        return fmt.Errorf("failed to perform first-pass unmarshal, error: %v", err)
    }

    // Next, extract the count from the mapping; if this fails return an error
    if err := extractValue(mapped, "count", &resp.Count); err != nil {
        return err
    }

    // Extract the next URL from the mapping; if this fails return an error
    if err := extractValue(mapped, "next_url", &resp.NextUrl); err != nil {
        return err
    }

    // Extract the request ID from the mapping; if this fails return an error
    if err := extractValue(mapped, "request_id", &resp.RequestId); err != nil {
        return err
    }

    // Extract the status from the mapping; if this fails return an error
    if err := extractValue(mapped, "status", &resp.Status); err != nil {
        return err
    }

    // Now, extract the results array into a temporary variable; if this fails return an error
    var results []interface{}
    if err := extractValue(mapped, "results", &results); err != nil {
        return err
    }

    // Finally, iterate over each result and add it to the slice of results by attempting
    // to convert it to a Struct; if any of these fail to convert then return an error
    resp.Results = make([]*structpb.Struct, len(results))
    for i, result := range results {
        if value, err := structpb.NewStruct(result.(map[string]interface{})); err == nil {
            resp.Results[i] = value
        } else {
            return fmt.Errorf("failed to create struct from result %d, error: %v", i, err)
        }
    }

    return nil
}

// Helper function that attempts to extract a value from a standard mapping of interfaces
// and set a field with it if the types are compatible
func extractValue[T any](mapping map[string]interface{}, field string, value *T) error {
    if raw, ok := mapping[field]; ok {
        if inner, ok := raw.(T); ok {
            *value = inner
        } else {
            return fmt.Errorf("failed to set value %v to field %s (%T)", raw, field, *value)
        }
    }

    return nil
}

然后,在我的服务代码中,我修改了代码的解组部分,以使用 Struct 对象。此代码依赖于 mapstructure 包装:

func getData[T ~proto.Message](data []byte) ([]T, error) {

    var resp *ArrayRespone
    if err := json.Unmarshal(data, &resp); err != nil {
        return nil, err
    }
    
    items := make([]T, len(resp.Results))
    for i, result := range resp.Results {
        var item T
        if err := mapstructure.Decode(result.AsMap(), &item); err != nil {
            return nil, err
        }

        items[i] = item
    }

    return items, nil
}

只要您的所有字段都可以轻松地反序列化为 google.protobuf.Value 类型然而,对于我来说,情况并非如此,因为我会调用类型中的几个字段 getData 具有的自定义实现 取消聚合JSON 。所以,我实际上选择的解决方案是使用 字节 相反:

使用字节

对于这个实现,我不需要依赖任何导入的类型,因此消息本身更容易使用:

message ArrayRespone {
    int32 count = 1;
    string next_url = 2;
    string request_id = 3;
    bytes results = 4;
    string status = 5;
}

这仍然需要为 取消聚合JSON ,但该实现也更简单:

func (resp *ArrayRespone) UnmarshalJSON(data []byte) error {

    // First, deserialize the JSON into a mapping between key fields and values
    // If this fails then return an error
    var mapped map[string]*json.RawMessage
    if err := json.Unmarshal(data, &mapped); err != nil {
        return fmt.Errorf("failed to perform first-pass unmarshal, error: %v", err)
    }

    // Next, extract the count from the mapping; if this fails return an error
    if err := extractValue(mapped, "count", &resp.Count); err != nil {
        return err
    }

    // Extract the next URL from the mapping; if this fails return an error
    if err := extractValue(mapped, "next_url", &resp.NextUrl); err != nil {
        return err
    }

    // Extract the request ID from the mapping; if this fails return an error
    if err := extractValue(mapped, "request_id", &resp.RequestId); err != nil {
        return err
    }

    // Extract the status from the mapping; if this fails return an error
    if err := extractValue(mapped, "status", &resp.Status); err != nil {
        return err
    }

    // Finally, iterate over each result and add it to the slice of results by attempting
    // to convert it to a Struct; if any of these fail to convert then return an error
    if raw, ok := mapped["results"]; ok {
        resp.Results = *raw
    }

    return nil
}

// Helper function that attempts to extract a value from a standard mapping of interfaces
// and set a field with it if the types are compatible
func extractValue[T any](mapping map[string]*json.RawMessage, field string, value *T) error {
    if raw, ok := mapping[field]; ok {
        if err := json.Unmarshal(*raw, &value); err != nil {
            return fmt.Errorf("failed to set value %s to field %s (%T)", *raw, field, *value)
        }
    }

    return nil
}

然后,我修改了 getData 函数为:

func getData[T ~proto.Message](data []byte) ([]T, error) {

    var resp *ArrayRespone
    if err := json.Unmarshal(data, &resp); err != nil {
        return nil, err
    }
    
    var items []T
    if err := json.Unmarshal(resp.Results, &items); err != nil {
        return nil, err
    }

    return items, nil
}

显然,这种实现更简单,并且需要更少的反序列化步骤,这意味着比 Struct 实施