The word marshaling means converting from a Go data type to an encoding, and unmarshaling means converting to a Go data type.
Marshalling可译作集结、结集、编码、编组、编集、安整、数据打包等，是计算机科学中把一个对象的内存表示变换为适合存储或发送的数据格式的过程。典型用于数据必须在一个程序的两个部分之间移动，或者必须从一个程序移动到另一个程序。Marshalling类似于序列化，可用于一个对象与一个远程对象通信。逆过程被称作unmarshalling。
Use Struct Tags To Add Metadata
Let’s say that we are building an order management system and have to read and write the following JSON:
{
"id":"12345",
"date_ordered":"2020-05-01T13:01:02Z",
"customer_id":"3",
"items":[{"id":"xyz123","name":"Thing 1"},{"id":"abc789","
name":"Thing 2"}]
}
We define types to map this data:
type Order struct {
ID string `json:"id"`
DateOrdered time.Time `json:"date_ordered"`
CustomerID string `json:"customer_id"`
Items []Item `json:"items"`
}
type Item struct {
ID string `json:"id"`
Name string `json:"name"`
}
We specify the rules for processing our JSON with struct tags, a string that is written after a field in a struct.
Even though struct tags are strings marked with backticks, they cannot extend past a single line.
Struct tags are composed of one or more tag/value pairs, written as tagName:"tagValue" and separated by spaces.
Because they are just strings, the compiler cannot validate that they are formatted correctly, but go vet does.
Also note that all of these field are exported. Like any other package, the code in the encoding/json package cannot access an unexported field on a struct in another package.
For JSON processing, we use the tag name json to specify the name of the JSON field that should be associated with the struct field.
If no json tag is provided, the default behavior is to assume that the name of the JSON object field matches the name of the Go struct field.
Despite this default behavior, it’s best to use the struct tag to specify the name for the field explicitly, even if it is identical to the field name.
When unmarshaling from JSON into a struct field with no json tag, the name match is case-insensitive.
When marshaling a struct field with no json tag back to JSON from a struct, the JSON field will always have an upper-case first letter, because the field is exported.
If a field should be ignored when marshaling or unmarshaling, use - for the name. If the field should be left out of the output when it is empty, add ,omitempty after the name.
Unmarshaling and Marshaling
The Unmarshal function in the encoding/json package is used to convert a slice of bytes into a struct. If we have a string named data, this is the code to convert data to a struct of type Order :
var o Order
err := json.Unmarshal([] byte( data), & o )
if err != nil {
return err
}
json.Unmarshal populates data into an input parameter, just like the implementations of the io.Reader interface.
There are two reasons for this.
First, just like io.Reader implementations, this allows for efficient re-use of the same struct over and over, giving you control over memory usage.
Second, there’s simply no other way to do it. Because Go doesn’t currently have generics, there’s no way to specify what type should be instantiated to store the bytes being read.
Even when Go adopts generics, the memory usage advantages will remain. We use the Marshal function in the encoding/json package to write an Order instance back as JSON, stored in a slice of bytes:
out,err :=json.Marshal(o)
This leads to the question: how are you able to evaluate struct tags? You might also be wondering how json.Marshal and json.Unmarshal are able to read and write a struct of any type. After all, every other method that we’ve written has only worked with types that were known when the program was compiled (even the types listed in a type switch are enumerated ahead of time).
The answer to both questions is reflection.
json.Encoder & json.Decoder
The json.Marshal and json.Unmarshal functions work on slices of bytes. As we just saw, most data sources and sinks in Go implement the io.Reader and io.Writer interfaces.
因为，大部分的源流 和 下游流，都是实现了io.Reader and io.Writer interfaces. 所以，如果我们想要在Go程序中，实现从源流取数据，处理后的结果，发送到下游流中。那么，go程序必须要有能力，直接从源流读取数据，也必须有能力，直接将处理好的结果，直接发送到下游流中。
而这个能力，就是本章需要讨论的json.Encoder & json.Decoder
// NewEncoder returns a new encoder that writes to w.
func NewEncoder(w io.Writer) *Encoder {
return &Encoder{w: w, escapeHTML: true}
}
json.Encoder encoder顾名思义，就是将数据进行编码打包，然后，写入到磁盘 或者 网络中。
因此，encoder其实就需要做2件事情：1. 将go strcut field，打包成json格式的数据 2. 将json数据，写到输出流中
常见的encode方式有：json、yaml、xml等。这里说的encode，是指将go struct field 转为 json，并不是说，将go struct field直接转为字节流数据
The encoding/json package includes two types that allow us to handle these situations.
The json.Encoder types write to anything that meets the io.Writer interfaces
The json.Decoder types read from anything that meets the io.Reader interfaces
The os.File type implements both the io.Reader and io.Writer interfaces, so we can use it to demonstrate json.Decoder and json.Encoder .
First, we write toFile to a temp file by passing the temp file to json.NewEncoder, which returns a json.Encoder for the temp file.
We then pass zhangsanPerson to the Encode method:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"os"
)
func main() {
zhangsanPerson := Person{
Name: "zhangsan",
Age: 10,
}
fmt.Println(os.TempDir())
tmpFile, err := ioutil.TempFile(os.TempDir(), "sample-")
if err != nil {
panic(err)
}
defer os.Remove(tmpFile.Name())
// 这一步，就是创建一个*Encoder，通过*Encoder，将go strcut field，转成json，并且写到磁盘中
err = json.NewEncoder(tmpFile).Encode(zhangsanPerson)
if err != nil {
panic(err)
}
err = tmpFile.Close()
if err != nil {
panic(err)
}
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
输出
C:\Users\weipeng.b\AppData\Local\Temp
接下来，我们再看一下json.Decoder
json.Decoder decoder顾名思义，就是读取字节流数据，然后进行解码，最终变成一个go struct field。
因此，decoder其实就需要做2件事情：1. 读取字节流数据 2. 将json数据，转成go struct field
The json.Decoder types read from anything that meets the io.Reader interfaces
下面，再看一个json.Decoder的示例
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"os"
)
func main() {
zhangsanPerson := Person{
Name: "zhangsan",
Age: 10,
}
fmt.Println(os.TempDir())
tmpFile, err := ioutil.TempFile(os.TempDir(), "sample-")
if err != nil {
panic(err)
}
defer os.Remove(tmpFile.Name())
// 这一步，就是创建一个*Encoder，通过*Encoder，将go strcut field，转成json，并且写到磁盘中
err = json.NewEncoder(tmpFile).Encode(zhangsanPerson)
if err != nil {
panic(err)
}
err = tmpFile.Close()
if err != nil {
panic(err)
}
tmpFile2, err := os.Open(tmpFile.Name())
if err != nil {
panic(err)
}
var readFromFilePerson Person
// 重点，在这里，解码后的数据，最终，会保存在readFromFilePerson中
err = json.NewDecoder(tmpFile2).Decode(&readFromFilePerson)
tmpFile2.Close()
if err != nil {
panic(err)
}
fmt.Println(readFromFilePerson)
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
输出
C:\Users\weipeng.b\AppData\Local\Temp
{zhangsan 10}
序列化 、json、字节流数组的关系
Encoding and Decoding JSON Streams
先介绍2个方法：
func (dec *Decoder) More() bool
More reports whether there is another element in the current array or object being parsed.
注意，这个方法，返回的bool值，表示，是否当前的数组 过或者对象中，是否还有其他的元素，等待被解析的。这里的元素，指的一定是json数据，中括号 和 大括号，不算作元素。
func (*Decoder) Token
func (dec *Decoder) Token() (Token, error)
Token returns the next JSON token in the input stream. At the end of the input stream, Token returns nil, io.EOF.
Token guarantees that the delimiters [ ] { } it returns are properly nested and matched: if Token encounters an unexpected delimiter in the input, it will return an error.
The input stream consists of basic JSON values—bool, string, number, and null—along with delimiters [ ] { } of type Delim to mark the start and end of arrays and objects. Commas and colons are elided.
这个方法，就是返回json中的一个个字符，包括中括号，大括号，字符串，数字，bool值等，但是会自动忽略掉 逗号 和 冒号
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"strings"
)
func main() {
const jsonStream = `
{"Message": "Hello", "Array": [1, 2, 3], "Null": null, "Number": 1.234}
`
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
t, err := dec.Token()
if err == io.EOF {
break
}
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v", t, t)
if dec.More() {
fmt.Printf(" (more)")
}
fmt.Printf("\n")
}
}
输出
json.Delim: { (more)
string: Message (more)
string: Hello (more)
string: Array (more)
json.Delim: [ (more)
float64: 1 (more)
float64: 2 (more)
float64: 3
json.Delim: ] (more)
string: Null (more)
<nil>: <nil> (more)
string: Number (more)
float64: 1.234
json.Delim: }
有了上面，2个方法的基础，我们再继续看下 decode stream
啥叫 decode stream？其实，就是，当我们有多个json对象后，如何使用流式编码的方式，每次解析一个json对象，但是可以连续解析多个json对象。这个就用到了Decoder.More方法
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
)
func main() {
const jsonStream = `
[
{"Name": "Ed", "Text": "Knock knock."},
{"Name": "Sam", "Text": "Who's there?"},
{"Name": "Ed", "Text": "Go fmt."},
{"Name": "Sam", "Text": "Go fmt who?"},
{"Name": "Ed", "Text": "Go fmt yourself!"}
]
`
type Message struct {
Name, Text string
}
dec := json.NewDecoder(strings.NewReader(jsonStream))
// read open bracket
t, err := dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v\n", t, t)
// while the array contains values
for dec.More() {
var m Message
// decode an array value (Message)
err := dec.Decode(&m)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%v: %v\n", m.Name, m.Text)
}
// read closing bracket
t, err = dec.Token()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%T: %v\n", t, t)
}
输出
json.Delim: [
Ed: Knock knock.
Sam: Who's there?
Ed: Go fmt.
Sam: Go fmt who?
Ed: Go fmt yourself!
json.Delim: ]
再来看另一个例子
package main
import (
"encoding/json"
"fmt"
"strings"
)
func main() {
const jsonStream = `
[
{"name": "Fred", "age": 40},
{"name": "Mary", "age": 21},
{"name": "Pat", "age": 30}
]
`
dec := json.NewDecoder(strings.NewReader(jsonStream))
token, err := dec.Token()
if err != nil{
panic(err)
}
fmt.Println(token)
fmt.Println("-----------------")
var t Person
for dec.More() {
err := dec.Decode(&t)
if err != nil {
panic(err)
}
fmt.Println(t)
}
fmt.Println("-----------------")
token, err = dec.Token()
if err != nil{
panic(err)
}
fmt.Println(token)
}
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
输出
[
-----------------
{Fred 40}
{Mary 21}
{Pat 30}
-----------------
]
下面，再看看encode json stream
package main
import (
"bytes"
"encoding/json"
"fmt"
)
func main() {
var b bytes.Buffer
encoder := json.NewEncoder(&b)
allInputSlice := make([]string, 0, 5)
allInputSlice = append(allInputSlice, "zhangsan")
allInputSlice = append(allInputSlice, "lisi")
allInputSlice = append(allInputSlice, "wanger")
for _,element := range allInputSlice {
err := encoder.Encode(element)
if err != nil {
panic(err)
}
}
out := b.String()
fmt.Println(out)
}
输出
"zhangsan"
"lisi"
"wanger"