Differences

This shows you the differences between two versions of the page.

--- tutorial:codec [2024/05/28 09:35] – solidblock
+++ tutorial:codec [2024/06/30 14:16] (current) – solidblock
@@ Line 1: / Line 1: @@
-====== Using codecs (DRAFT) ======
+====== Codecs ======
 ===== What is a codec =====
-A **codec**, introducted in Java Edition 1.16, is a specification of conversion between any type of object (such as ''LootTable'', ''Advancement'' or ''BlockPos'') and any type serialized from (nbt, json, etc.). A codec is a combination of encoder and decoder. An **encoder** encodes an object to serialized form, and a **decoder** decodes the serialized from into objects.
+A **codec**, introduced in Java Edition 1.16, is a specification of conversion between any type of object (such as ''LootTable'', ''Advancement'' or ''BlockPos'') and any serialized form (nbt, json, etc.). A codec is a combination of encoder and decoder. An **encoder** encodes an object to serialized form, and a **decoder** decodes the serialized from into objects.
-For example, loot tables are written in json forms in data packs, and a loaded as ''LootTable'' objects in the server. To load loot tables from the jsons in the data pack, the codec is used. There are many pre-written codecs in Minecraft, each of which, is used for a specific type of object, but can serialize or deserialize between it and different types of serialized from. For example, ''LootTable.CODEC'' can convert ''LootTable'' objects into jsons and nbts, and can also convert jsons or nbts into ''LootTable'' objects.
+For example, loot tables are written in json forms in data packs, and are loaded as ''LootTable'' objects in the server. To load loot tables from the jsons in the data pack, the codec is used, to decode from json files to ''LootTable'' objects. There are many pre-written codecs in Minecraft, each of which, is used for a specific type of object, but can serialize or deserialize between it and different types of serialized form. For example, ''LootTable.CODEC'' can convert ''LootTable'' objects into jsons and nbts, and can also convert jsons or nbts into ''LootTable'' objects.
-Codec was introduced in 1.16, but has been in increasingly widely used in Minecraft since 1.20. For example, ''LootTable'', ''Advancements'' and ''Text'', previously serialized or deserialized in other manners in older versions, but now using Codec. Item components, introducted in 1.20.5, also use codecs to serialize.
+Codec was introduced in 1.16, but has mainly been increasingly widely used in Minecraft since 1.20. For example, ''LootTable'', ''Advancements'' and ''Text'', are previously serialized or deserialized in other manners in older versions, but now using codecs. Item components, introduced in 1.20.5, also use codecs to serialize and deserialize.
-===== How to use a codec =====
+===== Using a codec =====
-When you serialize or deserialize, you need to specify a ''DynamicOps'', which specifies each concrete action in the serialization or deserialization. Mose common used are ''JsonOps.INSTANCE'' and ''NbtOps.INSTANCE''. The following code takes an example of converting between ''BlockPos'' and ''NbtElement''.
+When you serialize or deserialize, you need to specify a ''DynamicOps'', which specifies each concrete action in the serialization or deserialization. The most common used are ''JsonOps.INSTANCE'' and ''NbtOps.INSTANCE''. The following code shows an example of converting between ''BlockPos'' and ''NbtElement''.
 <code java>
     // serializing a BlockPos
@@ Line 25: / Line 25: @@
 </code>
-As seen in the code, the result is not directly ''NbtElement'' or ''BlockPos'', but a ''DataResult'' wrapping them. That's because errors are common in serialization, and instead of exceptions, errors in the data results often happens. You can fetch the result with ''result()'' (which may be empty when error happens), or fecth the error message with ''error()'' (which may be empty when error does not happen). You can also directly fetch the result with ''getOrThrow()'' (or ''Util.getResult'' in older versions).
+As seen in the code, the result is not directly ''NbtElement'' or ''BlockPos'', but a ''DataResult'' wrapping them. That's because errors are common in serialization, and instead of exceptions, errors in the data results often happens. You can fetch the result with ''result()'' (which may be empty when error happens), or fetch the error message with ''error()'' (which may be empty when error does not happen). You can also directly fetch the result with ''getOrThrow()'' (or ''Util.getResult'' in older versions).
 <code java>
-    // get the result when succeed
+    // get the result when succeeded
     final NbtList nbtList = new NbtList();
     nbtList.add(NbtInt.of(1));
@@ Line 41: / Line 41: @@
 </code>
-===== How to write a codec =====
+There is a special type of ''DynamicOps'', named ''RegistryOps'', which is usually created with ''RegistryWrapper.getOps''. It is a wrapper of other ops, which, when encoding or decoding, behaves basically identical to the wrapped ops, but has some differences: when encoding or decoding some registry entries, it will use the specified ''RegistryWrapper'' to get objects or entries, while other ops may directly use the registry or even throw an error.
+===== Writing a codec =====
+==== Vanilla existing codecs ====
+Mojang has already written many codecs for you. You can directly use them in some cases, and in complex codecs, they may be also useful.
+For primitive types, codecs are stored as fields of ''Codec''. For example:
+  * ''Codec.BOOLEAN'' is a codec for boolean.
+  * ''Codec.INT'' is a codec for int.
+  * ''Codec.FLOAT'' is a codec for float.
+For classes belonging to Minecraft, codecs are stored as their fields. For example:
+  * ''Identifier.CODEC'' is a codec for ''Identifier''.
+  * ''BlockPos.CODEC'' is a codec for ''BlockPos''.
+  * ''LootTable.CODEC'' is a codec for ''LootTable''.
+Besides, ''Codecs'' provdes from utilities for codecs. For example:
+  * ''Codecs.JSON_ELEMENT'' is a codec for ''JsonElement''.
+  * ''Codecs.NOT_EMPTY_STRING'' is a codec for ''String''. Similar to ''Codec.STRING'', but throws an error when the string is empty.
+  * ''Codecs.rangedInt(1, 5)'' is a codec for ''Integer'', but throws an error when the integer is not within range ''[1, 5]''.
+> **Tips:** You can learn more about how to write codecs by seeing how vanilla codecs are written.
 ==== Mapping existing codec ====
-For simple objects, you can directly convert it between those have existing codecs. Mojang provides codecs for all primitive types and common types. In this example, you want to create a codec for ''Identifier''. We just know that it can be converted from and to a ''String'', then we map like this:
+For simple objects, you can directly convert it between those have existing codecs. Mojang provides codecs for all primitive types and common types. In this example, you want to create a codec for ''Identifier''. We just know that it can be converted from and to a ''String'', then we map through **''xmap''**:
 <code java>
-    Codec<Identifier> identifierCodec = Codec.STRING.xmap(s -> new Identifier(s), identifier -> identifier.toString());
+    // replace ''Identifier.of'' with ''new Identifier'' for versions before 1.21.
+    Codec<Identifier> identifierCodec = Codec.STRING.xmap(s -> Identifier.of(s), identifier -> identifier.toString());
 </code>
 When decoding, the serialized from is converted into string through ''Codec.STRING'', and then converted into ''Identifier'' through the first lambda. When encoding, the ''Identifier'' is converted into string through the second lambda, and then encoded through ''Codec.STRING''.
-If the serialized from is something that cannot be converted into a string, a codec result may be ''DataResult.Error'', and handled properly as expected. However, if you pass a string that cannot be an Identifier, such as ''NbtString.of("ABC")'', decoding it will directly throw ''InvalidIdentifierException''. Therefore, to handle errors properly, we write like this:
+If the serialized from is something that cannot be converted into a string, a codec result may be ''DataResult.Error'', and handled properly as expected. However, if you pass a string that cannot be an Identifier, such as ''NbtString.of("ABC")'', decoding it will directly throw ''InvalidIdentifierException''. Therefore, to handle errors properly, we use **''flatXmap''**:
 <code java>
     Codec<Identifier> identifierCodec = Codec.STRING.flatXmap(s -> {
       try {
-        return DataResult.success(new Identifier(s));
+        // replace ''Identifier.of'' with ''new Identifier'' for versions before 1.21.
+        return DataResult.success(Identifier.of(s));
       } catch (InvalidIdentifierException e) {
         return DataResult.error(() -> "The identifier is invalid:" + e.getMessage());
@@ Line 65: / Line 88: @@
 </code>
-Note that this time we use ''flatXmap'' instead of ''xmap'', in which, the two lambdas returns ''DataResult''. This is used then two lambdas may return failed results. In this case, the second lambda does not fail, so we can also write like this:
+Note that this time we use ''flatXmap'' instead of ''xmap'', in which, the two lambdas returns ''DataResult''. This is used then two lambdas may return failed results. In this case, the second lambda does not fail, so we can also use **''comapFlatMap''**:
 <code java>
     Codec<Identifier> identifierCodec = Codec.STRING.comapFlatMap(s -> {
       try {
-        return DataResult.success(new Identifier(s));
+        // replace ''Identifier.of'' with ''new Identifier'' for versions before 1.21.
+        return DataResult.success(Identifier.of(s));
       } catch (InvalidIdentifierException e) {
         return DataResult.error(() -> "The identifier is invalid:" + e.getMessage());
@@ Line 75: / Line 99: @@
     }, identifier -> identifier.toString());
 </code>
+==== Record codec ====
+=== Required fields ===
+Most objects are complicated, which cannot simply be represented in primitive type. It may be in a form of a map-like object with multiple fields, such as ''NbtCompound'' or ''JsonObject''. In this case, you need to specify the fields, including name and codecs of the fields. For example, we create a record type:
+<code java>
+public record Student(String name, int id, Vec3d pos) {
+}
+</code>
+To create a codec for this record, we should specify the fields codec, specify how to get fields from a ''Student'' object, and specify how to create a ''Student'' object from the fields:
+<code java>
+  public static final Codec<Student> CODEC = RecordCodecBuilder.create(i -> i.group(
+      Codec.STRING.fieldOf("name").forGetter(Student::name),
+      Codec.INT.fieldOf("id").forGetter(Student::id),
+      Vec3d.CODEC.fieldOf("pos").forGetter(Student::pos)
+  ).apply(i, Student::new));
+</code>
+In this example, the method ''group'' takes three fields, specifying each field of the record. The parameter of ''fieldOf'' is the name of the field, which represents as keys of the encodec result. The method ''forGetter'' specifies how to get the value of the field from the object, such as ''student -> student.name()'' (in lambda form) or ''Student::name'' (in method reference form). The second parameter of ''apply'' specified how to convert the field values to a complete object, such as ''(string, i, vec3d) -> new Student(string, i, vec3d)''.
+Let's demonstrate the effect of the codec:
+  * ''%%new Student("Steve", 20, Vec3d.ZERO)%%'' will be encoded as ''{name: Steve, id: 20, pos: [0, 0, 0]}''.
+  * ''%%new Student("Alex", 25, new Vec3d(1d, 2d, 3d))%%'' will be encoded as ''{name: Alex, id: 25, pos: [1d, 2d, 3d]}''.
+  * ''{name: Steve, id: 20, pos: [0, 0, 0]}'' will be decoded as ''%%new Student("Steve", 20, Vec3d.ZERO)%%''.
+  * ''{name: Steve, id: 30}'' cannot be decoded, because it misses a required field ''pos''.
+=== Optional fields ===
+Sometimes all fields are not required. In the previous example, if the encoded result misses some fields, error will be thrown. To make the fields optional, use ''optionalFieldOf''. You also need to specify a default value:
+<code java>
+  public static final Codec<Student> CODEC = RecordCodecBuilder.create(i -> i.group(
+      Codec.STRING.fieldOf("name").forGetter(Student::name),
+      Codec.INT.optionalFieldOf("id", 0).forGetter(Student::id),
+      Vec3d.CODEC.fieldOf("pos").forGetter(Student::pos)
+  ).apply(i, Student::new));
+</code>
+In this case, when decoding, when the field ''id'' does not exist, the default value ''0'' will be taken. When encoding, when the field equals to default value ''0'', it will be missing in the result. For example:
+  * ''%%new Student("Steve", 0, Vec3d.ZERO)%%'' will be encoded as ''{name: Steve, pos: [0, 0, 0]}''.
+  * ''{name: Steve, pos: [0, 0, 0]}'' will be decoded as ''%%new Student("Steve", 0, Vec3d.ZERO)%%''.
+  * Of course, ''{name: Steve, id: 0, pos: [0, 0, 0]}'' will be also decoded as ''%%new Student("Steve", 0, Vec3d.ZERO)%%''.
+  * ''{name: Steve, id: Hello, pos: [0, 0, 0]}'' cannot be decoded, because the field ''id'' is invalid.
+Pay attention to the last example. When decoding, when an optional field has an invalid value, error will be thrown. Howevevr, in older Minecraft versions, when an optional field has an invalid value, the default value will be directly taken.
+  * In older Minecraft versions, ''{name: Steve, id: Hello, pos: [0, 0, 0]}'' will be encoded as ''%%new Student("Steve", 0, Vec3d.ZERO)%%''.
+> **Note:** In current versions, you can also replace ''optionalFieldOf'' with ''lenientOptionalFieldOf'', so as to take default values when the value is invalid.
+If you do not provide a default value for ''optionalFieldOf'' or ''lenientOptionalFieldOf'', it will be a field codec for ''Optional<T>''. For example, if the ''name'' of a ''Student'' is optional:
+<code java>
+public record Student(Optional<String> name, int id, Vec3d pos) {
+  public static final Codec<Student> CODEC = RecordCodecBuilder.create(i -> i.group(
+      Codec.STRING.optionalFieldOf("name").forGetter(Student::name),
+      Codec.INT.fieldOf("id").forGetter(Student::id),
+      Vec3d.CODEC.fieldOf("pos").forGetter(Student::pos)
+  ).apply(i, Student::new));
+}
+</code>
+=== Another from of field codec ===
+The codec can also be written like this:
+<code java>
+public record Student(String name, int id, Vec3d pos) {
+  public static final Codec<Student> CODEC = RecordCodecBuilder.create(i -> i.apply3(
+      Student::new,
+      Codec.STRING.fieldOf("name").forGetter(Student::name),
+      Codec.INT.fieldOf("id").forGetter(Student::id),
+      Vec3d.CODEC.fieldOf("pos").forGetter(Student::pos)
+  ));
+}
+</code>
+Writing like this may be simpler. You need to specify the method to convert fields to an objects at first (''Student::new'' in this example), and then write field codecs following it, with IDEs able to properly provide suggestions. Remember the method name (''apply3'' in this example) should contain the number of fields in this situation. For example, sometimes you may also use ''apply4'', ''apply5'', etc. If your record codec contains only one field, use ''ap''.
+==== Dispatching codec ====
+Some objects may not be in fixed structures, but have variant types, each of which, is in a unique structure. Therefore, dispatching codecs are used.
+When encoding, the codec gets the type from the object, encode it according to a codec corresponding to the "type", then encode the type, and finally add the type as a field ''type'' of the serialized map-like object. When decoding, obtains and decodes the type from the ''type'' field of the map-like object, and then decode it according to the codec corresponding to the type.
+Let's take this example: ''SchoolMember'' has three types: ''Student'', ''Teacher'' and ''Staff'', each of which has a unique structure.
+<code java>
+public interface SchoolMember {
+  record Student(String name, int id) implements SchoolMember {}
+  record Teacher(String name, String subject, int id) implements SchoolMember {}
+  record Staff(String name, String department, Identifier id) implements SchoolMember {}
+}
+</code>
+It's easy to know that each type can be created a specific codec (note that it is ''MapCodec'', not ''Codec''):
+<code java>
+public interface SchoolMember {
+  record Student(String name, int id) implements SchoolMember {
+    public static final MapCodec<Student> CODEC = RecordCodecBuilder.mapCodec(i -> i.group(Codec.STRING.fieldOf("name").forGetter(Student::name), Codec.INT.fieldOf("id").forGetter(Student::id)).apply(i, Student::new));
+  }
+  // Codecs of other two types are omitted here.
+}
+</code>
+We also need to specify a serializable "type" itself. In many cases (such as vanilla ''LootTable''), the type is based on identifier, but in this case we simply use enum.
+<code java>
+public interface SchoolMember {
+  @NotNull Type getType();
+  enum Type implements StringIdentifiable {
+    STUDENT("student"), TEACHER("teacher"), STAFF("staff");
+    public static final Codec<Type> CODEC = StringIdentifiable.createCodec(Type::values);
+    private final String name;
+    Type(String name) {
+      this.name = name;
+    }
+    @Override
+    public String asString() {
+      return name;
+    }
+  }
+  record Student(String name, int id) implements SchoolMember {
+    public static final Codec<Student> CODEC = RecordCodecBuilder.create(i -> i.group(Codec.STRING.fieldOf("name").forGetter(Student::name), Codec.INT.fieldOf("id").forGetter(Student::id)).apply(i, Student::new));
+    @Override
+    public @NotNull Type getType() {
+      return Type.STUDENT;
+    }
+  }
+  // The other two types are omitted here.
+}
+</code>
+Now we specified how to get the type from the object. However, it is also required to specify how to get the codec from the type. It can be achieved in many ways, such as storing fields in the type object, or using a map. But in this example, for simplicity, we just use a simple "switch" statements. Then, to create a dispatching codec, invoke method ''**dispatch**'' to the codec of the type:
+<code java>
+  Codec<SchoolMember> CODEC = Type.CODEC.dispatch(SchoolMember::getType, type -> switch (type) {
+    case STAFF -> Staff.CODEC;
+    case STUDENT -> Student.CODEC;
+    case TEACHER -> Teacher.CODEC;
+  });
+</code>
+If you need to specify another name of the field, add a string as a first parameter of ''dispatch'' method.
+Let's see the effect our example:
+  * ''%%new SchoolMember.Student("Steve", 15)%%'' will be encoded as ''{type: student, name: Steve, id: 15}''.
+  * ''{type: teacher, name: Alex, department: chemistry, id: 18}'' will be decoded as ''%%new SchoolMember.Teacher("Alex", "chemistry", 18)%%''.
+> **Note:** Actually, in practice, the types can be more complicated. Therefore, you may use registry for types, such as vanilla ''LootFunctionType'', which has a specific registry ''Registries.LOOT_FUNCTION_TYPE''. It is more flexible and extendable.
+===== Packet codec =====
+A **packet codec**, different from codec, converts between objects and binery packets. It is sometimes similar to codec, and also used in many cases such as item components, but for complex objects, it uses //tuples// instead of //maps//. Packet codecs for primitive types are stored in ''PacketCodecs''. We can also use ''PacketCodec.of'' to directly specify encoding and decoding. For example:
+<code java>
+public record Student(String name, int id, Vec3d pos) {
+  public static final PacketCodec<PacketByteBuf, Student> PACKET_CODEC = PacketCodec.tuple(
+      PacketCodecs.STRING, Student::name,
+      PacketCodecs.INTEGER, Student::id,
+      PacketCodec.of(
+          // encoder: writing to the packet
+          (value, buf) -> buf.writeDouble(value.x).writeDouble(value.y).writeDouble(value.z),
+          // decoder: reading the packet
+          buf -> new Vec3d(buf.readDouble(), buf.readDouble(), buf.readDouble())
+      ), Student::pos,
+      Student::new);
+}
+</code>
+> **Note:** Besides ''PacketByteBuf'', you may also sometimes see ''RegistryByteBuf''. Similar to ''RegistryOps'' explained before, it works the same as ''PacketByteBuf'', but also provides a ''registryManager'' so as to obtain some registry elements.