在neuraxle 文檔中顯示了一個示例，使用存儲庫在管道中延遲加載數據，請參閱以下代碼：from neuraxle.pipeline import Pipeline, MiniBatchSequentialPipelinefrom neuraxle.base import ExecutionContextfrom neuraxle.steps.column_transformer import ColumnTransformerfrom neuraxle.steps.flow import TrainOnlyWrappertraining_data_ids = training_data_repository.get_all_ids()context = ExecutionContext('caching_folder').set_service_locator({ BaseRepository: training_data_repository})pipeline = Pipeline([ ConvertIDsToLoadedData().assert_has_services(BaseRepository), ColumnTransformer([ (range(0, 2), DateToCosineEncoder()), (3, CategoricalEnum(categeories_count=5, starts_at_zero=True)), ]), Normalizer(), TrainOnlyWrapper(DataShuffler()), MiniBatchSequentialPipeline([ Model() ], batch_size=128)]).with_context(context)但是，它沒有顯示如何實現BaseRepository和ConvertIDsToLoadedData類。實施這些課程的最佳方式是什么？誰能舉個例子嗎？
1 回答
慕勒3428872
TA貢獻1848條經驗 獲得超6個贊
我沒有檢查以下是否編譯，但它應該如下所示。如果您發現需要更改的內容并嘗試編譯它，請有人編輯此答案：
class BaseDataRepository(ABC):?
? ? @abstractmethod
? ? def get_all_ids(self) -> List[int]:?
? ? ? ? pass
? ? @abstractmethod
? ? def get_data_from_id(self, _id: int) -> object:?
? ? ? ? pass
class InMemoryDataRepository(BaseDataRepository):?
? ? def __init__(self, ids, data):?
? ? ? ? self.ids: List[int] = ids
? ? ? ? self.data: Dict[int, object] = data
? ? def get_all_ids(self) -> List[int]:?
? ? ? ? return list(self.ids)
? ? def get_data_from_id(self, _id: int) -> object:?
? ? ? ? return self.data[_id]
class ConvertIDsToLoadedData(BaseStep):?
? ? def _transform_data_container(self, data_container: DataContainer, context: ExecutionContext):?
? ? ? ? repo: BaseDataRepository = context.get_service(BaseDataRepository)
? ? ? ? ids = data_container.data_inputs
? ? ? ? # Replace data ids by their loaded object counterpart:?
? ? ? ? data_container.data_inputs = [repo.get_data_from_id(_id) for _id in ids]
? ? ? ? return data_container, context
context = ExecutionContext('caching_folder').set_service_locator({
? ? BaseDataRepository: InMemoryDataRepository(ids, data)? # or insert here any other replacement class that inherits from `BaseDataRepository` when you'll change the database to a real one (e.g.: SQL) rather than a cheap "InMemory" stub.?
})
添加回答
舉報
0/150
提交
取消