Serde¶
Bigflow 内置序列化/反序列化器
-
class
bigflow.serde.
CPickleSerde
¶ -
使用CPickle去做序列化/反序列化的Serde
-
class
bigflow.serde.
ChainSerde
(*args)¶ -
serde for chain transform. for example:
serdex should deserialize a string unless the last serde. ChainSerde(serdeA, serdB); serdeB->serialize(serdeA->serialize(object,obj)); serdeA->deserialize(serdeB->deserialize(buf));-
deserialize
(buf)¶
-
get_args
()¶
-
serialize
(obj)¶
-
-
class
bigflow.serde.
DefaultSerde
¶ 基类:
bigflow.core.serde.omnitypes_objector.OmniObjector
,bigflow.serde.CppSerde
默认的序列化/反序列化器
-
class
bigflow.serde.
FastPbSerdeFactory
(pipeline, proto_file)¶ 基类:
object
Create ProtobufSerde by fast-python-pb
>>> _serde_factory = serde.FastPbSerdeFactory(_pipeline, './sample.proto') >>> _serde = _serde_factory.get_serde('bigflow.MESSAGE') >>> data = _pipeline.read(input.SequenceFile(input_path, serde=_serde))
-
get_serde
(class_name)¶ get protobuf serde
-
-
class
bigflow.serde.
IdlPacketSerde
(log_type)¶ -
Serde for idl_packet
-
deserialize
(buf)¶
-
get_log_type
()¶ get log_type
-
serialize
(obj)¶ serialzie
-
-
class
bigflow.serde.
ObjectorSerde
(name, config='')¶ -
Serde for objector serde
-
get_entity_config
()¶
-
get_entity_name
()¶ 获取类名
-
-
class
bigflow.serde.
Optional
(serde)¶ -
用于支持None的序列化/反序列化
-
deserialize
(buf)¶
-
origin_serde
()¶
-
serialize
(obj)¶
-
-
class
bigflow.serde.
ProtobufSerde
(msg_type, raise_exception_when_failed=True)¶ -
Serde for protobuf
-
deserialize
(buf)¶
-
get_msg_type
()¶
-
serialize
(msg)¶
-
-
class
bigflow.serde.
SameTypeListSerde
(value)¶ -
list序列化/反序列化器,要求list中元素均为同一类型
-
deserialize
(buf)¶
-
serialize
(obj)¶
-
-
class
bigflow.serde.
Serde
¶ 基类:
bigflow.core.entity.EntitiedBySelf
所有python实现的serde类要求继承于此类,用户可以通过继承Serde实现自定义的序列化/反序列化器
>>> from bigflow import serde >>> >>> class CustomSerde(serde.Serde): ... def serialize(self, obj): ... assert(isinstance(obj, MyCostomObject)) ... return str(obj) ... def deserialize(self, buf): ... return MyCostomObject(buf)
在大部分的变换中都可以传递一个特殊的叫serde的参数, 设置该参数即可指定产出的PCollection的serde。
p.map(lambda x: MyCostomObject(x), serde=CustomSerde())-
get_entity_config
()¶
-
get_entity_name
()¶ 获取类名
-
-
class
bigflow.serde.
SetSerde
(value)¶ -
Serde for set
-
deserialize
(buf)¶
-
get_value_serde
()¶
-
serialize
(obj)¶
-
-
class
bigflow.serde.
TupleLikeListSerde
(*value)¶ -
list序列化/反序列化器,list中每个元素可以为任意类型
-
deserialize
(buf)¶
-
serialize
(obj)¶
-
-
bigflow.serde.
any
()¶ Return a serde use the default serde.
-
bigflow.serde.
bool_
()¶ Return an optional bool serde.
-
bigflow.serde.
common_serde
(*serdes)¶ 输出输入serde集合中的公共serde 目前只维护系统定义的serde, 对于用户自定义的serde, 返回None :param serdes: serde instances or classes
返回: 如果找到合适的公共serde则返回,否则返回None
-
bigflow.serde.
dict_of
(key, value)¶ Return an optional dict serde
-
bigflow.serde.
extract_elem
(serde, n, if_not_found=None)¶ inner function
-
bigflow.serde.
float_
()¶ Return an optional float serde.
-
bigflow.serde.
int_
()¶ Return an optional int serde.
-
bigflow.serde.
list_of
(*value)¶ Return an optional list serde.
-
bigflow.serde.
of
(rtype)¶ Return the serde you want.
If the input arg is a type instance, such as int, str, float or a tuple of type instances, such as (int, str) or a list of type instances, such as [int, str] and [int] the function will return the corresponding serde to serialize and deserialize the data.
The returned serde can process the Nonable data.
- Note: [int] means a list of ints, it can accept any number of ints you want,
- but [int, str] can only accecpt a list who has exactly 2 elements. (if your data to serialize has more than 2 elements, the remaining elements may be lost)
-
bigflow.serde.
origin
(serde)¶ inner function
-
bigflow.serde.
proto_of
(msg_type)¶ Return an protobuf serde Warning: this is not optional
-
bigflow.serde.
sample
(value)¶ Return a serde can process your input data.
Note: if the sample is a list, when all the data is the same type, we assume
-
bigflow.serde.
set_of
(value)¶ Return an optional set serde
-
bigflow.serde.
str_
()¶ Return an optional str serde.
-
bigflow.serde.
tuple_of
(*args)¶ Return an optional tuple serde.