What SDS solution do you recommend?

Hi @Evgeny Belenky ​, HERE ARE THE STORAGE REQUIREMENTS FOR DEEP LEARNING Deep learning workloads are a special kind of beast: all DL data is considered hot data, which raises the dilemma of not being able to employ any sort of tiered storage management solution. This is because normal SSDs usually used for hot data under conventional conditions simply won’t move the data required for millions, billions, or even trillions of metadata transfers for an ML training model to classify an unknown something out of only a limited amount of examples. Below are a few examples of a few storage requirements needed to avoid the dreaded curse of dimensionality. COST EFFICIENCY Enormous AI data sets become an even bigger burden if they don’t fall within the budget set aside for storage. Anyone who has been in charge of managing enterprise data for any amount of time knows well that highly-scalable systems have always been more high-priced on a capacity versus cost basis. The ultimate deep learning storage system must be both affordable and scalable to make sense. PARALLEL ARCHITECTURE In order to avoid those dreaded choke points that stunt a deep learning machine’s ability to learn, it’s essential for data sets t to have parallel-access architecture. DATA LOCALITY While it might be possible that many organizations may opt to keep some of their data on the cloud, most of it should remain on-site in a data center. There are at least three reasons for this: regulatory compliance, cost efficiency, and performance. For this reason, on-site storage must rival the cost of keeping it on the cloud. HYBRID ARCHITECTURE As touched on above, different types of data have unique performance requirements. Thus, storage solutions should offer the perfect mixture of storage technologies instead of an asymmetrical strategy that will eventually fail. It’s all about simultaneously meeting ML storage performance and scalability. SOFTWARE-DEFINED STORAGE Not all huge data sets are the same—especially in terms of DL and ML. While some of them can get by with the simplicity of pre-configured machines, others need hyper-scale data centers featuring purpose-built servers architectures that are previously set in place. This is what makes software-defined storage solutions the best option. Our X-AI Accelerated is an any–scale DL and ML solution that offers unmatched versatility for any organization’s needs. X-AI Accelerated was engineered from the ground up and optimized for “ingest, training, data transformations, replication, metadata, and small data transfers.” Not only that but RAID Inc. offers all the aforementioned requirements such as all-flash NVMe X2-AI/X4-AI or the X5-AI, which are hybrid flash and hard drive storage platforms. Both the NVMe X2-AI/X4-AI and the X5-AI support parallel access to flash and deeply expandable HDD storage as well. Furthermore, the X-AI Accelerated storage platform permits one to scale out from only a few TBs to tens of PBs.

Storage requirements are depending on several factors.  For example, the language you will use, the data you will receive, do you use a RAID, etc. 

SDS is very flexible and can be adapted to the most diverse requirements. Many solutions are able to provide storage via iSCSI, iSER, Fibre channel or NVMeOF. The storage services are offered across all connected storage sources (e.g. mirroring, replication, snapshots, thin provisioning, etc.) It is quite easy to implement new services or protocols. An SDS solution can be much cheaper to purchase, as standard servers are used. The disadvantage of an SDS can be the complexity. Many different sources can be combined into one solution. These sources can be managed differently, or incompatibilities can occur. The support of such an environment can then be very difficult. Another disadvantage is that performance and latency depend very much on the hardware used. This can be a limiting factor. Another disadvantage is that the manufacturer of the SDS has hardly any influence on the OS on which the solution runs, which can lead to problems, especially with updates and patches. Here it must be tested in advance whether the SDS software continues to work properly. But for me SDS is superior to any conventional solution. You can reach a goal easier, more flexible and cheaper.

Software-Defined Storage (SDS) enables us to utilize commodity x86 servers as reliable high performance, highly available storage. Before opting for an SDS Solution things you should consider. Reliable New servers which should last at least 5 years with an extended warranty. Design your SDS Solution properly considering RAID Selection, Disk Type Selection, Storage Network Design, Network Redundancy and Failover in mind. Designing Storage Network can be complex, and troubleshooting Storage network issues can be easier only if you document the whole setup. Make sure to add redundancy on every hardware and connectivity level i-e Disk level, Storage Node/Host level, Storage Connectivity Paths level. Pros: Very Cost-efficient Storage solution. Easier to switch between SDS Solutions, avoids vendor lock-in. Dissimilar and incompatible Underlying devices and Disks can be used to host Storage. Versatile protocol support. Implementing Redundancy is also cost effective. Cons: SDS may also require software-defined networking, it will be complex to design and maintain. Implementing an SDS solution requires hardware and software resources from multiple different vendors, which may incur compatibility issues.

very good and valuable information