# Software Stack

SINGA's software stack includes two major levels, the low level backend classes and the Python interface level. Figure 1 illustrates them together with the hardware. The backend components provide basic data structures for deep learning models, hardware abstractions for scheduling and executing operations, and communication components for distributed training. The Python interface wraps some CPP data structures and provides additional high-level classes for neural network training, which makes it convenient to implement complex neural network models.

Next, we introduce the software stack in a bottom-up manner.

**Figure 1 - SINGA V3
software stack.**

## Low-level Backend

### Device

Each `Device`

instance, i.e., a device, is created against one hardware device,
e.g. a GPU or a CPU. `Device`

manages the memory of the data structures, and
schedules the operations for executing, e.g., on CUDA streams or CPU threads.
Depending on the hardware and its programming language, SINGA have implemented
the following specific device classes:

**CudaGPU**represents an Nvidia GPU card. The execution units are the CUDA streams.**CppCPU**represents a normal CPU. The execution units are the CPU threads.**OpenclGPU**represents normal GPU card from both Nvidia and AMD. The execution units are the CommandQueues. Given that OpenCL is compatible with many hardware devices, e.g. FPGA and ARM, the OpenclGPU has the potential to be extended for other devices.

### Tensor

`Tensor`

class represents a multi-dimensional array, which stores model
variables, e.g., the input images and feature maps from the convolution layer.
Each `Tensor`

instance (i.e. a tensor) is allocated on a device, which manages
the memory of the tensor and schedules the (computation) operations against
tensors. Most machine learning algorithms could be expressed using (dense or
sparse) the tensor abstraction and its operations. Therefore, SINGA would be
able to run a wide range of models, including deep learning models and other
traditional machine learning models.

### Operator

There are two types of operators against tensors, linear algebra operators like
matrix multiplication, and neural network specific operators like convolution
and pooling. The linear algebra operators are provided as `Tensor`

functions and
are implemented separately for different hardware devices

- CppMath (tensor_math_cpp.h) implements the tensor operations using Cpp for CppCPU
- CudaMath (tensor_math_cuda.h) implements the tensor operations using CUDA for CudaGPU
- OpenclMath (tensor_math_opencl.h) implements the tensor operations using OpenCL for OpenclGPU

The neural network specific operators are also implemented separately, e.g.,

- GpuConvFoward (convolution.h) implements the forward function of convolution via CuDNN on Nvidia GPU.
- CpuConvForward (convolution.h) implements the forward function of convolution using CPP on CPU.

Typically, users create a `Device`

instance and use it to create multiple
`Tensor`

instances. When users call the Tensor functions or neural network
operations, the corresponding implementation for the resident device will be
invoked. In other words, the implementation of operators is transparent to
users.

The Tensor and Device abstractions are extensible to support a wide range of hardware device using different programming languages. A new hardware device would be supported by adding a new Device subclass and the corresponding implementation of the operators.

Optimizations in terms of speed and memory are done by the `Scheduler`

and
`MemPool`

of the `Device`

. For example, the `Scheduler`

creates a
computational graph according to the dependency of the operators.
Then it can optimize the execution order of the operators for parallelism and
memory sharing.

### Communicator

`Communicator`

is to support distributed training. It implements
the communication protocols using sockets, MPI and NCCL. Typically users only
need to call the high-level APIs like `put()`

and `get()`

for sending and
receiving tensors. Communication optimization for the topology, message size,
etc. is done internally.

## Python Interface

All the backend components are exposed as Python modules via SWIG. In addition, the following classes are added to support the implementation of complex neural networks.

### Opt

`Opt`

and its subclasses implement the methods (such as SGD) for updating model
parameter values using parameter gradients. A subclass DistOpt
synchronizes the gradients across the workers for distributed training by
calling methods from `Communicator`

.

### Operator

`Operator`

wraps multiple functions implemented using the Tensor or neural
network operators from the backend. For example, the forward function and
backward function `ReLU`

compose the `ReLU`

operator.

### Layer

`Layer`

and its subclasses wraps the operators with parameters. For instance,
convolution and linear operators

have weight and bias parameters. The parameters are maintained by the
corresponding `Layer`

class.

### Autograd

Autograd implements the
reverse-mode automatic differentiation
by recording the execution of the forward functions of the operators calling the
backward functions automatically in the reverse order. All functions can be
buffered by the `Scheduler`

to create a computational graph for
efficiency and memory optimization.

### Model

Model provides an easy interface to implement new network models. You
just need to inherit `Model`

and define the forward propagation of the model by
creating and calling the layers or operators. `Model`

will do autograd and
update the parameters via `Opt`

automatically when training data is fed into it.
With the `Model`

API, SINGA enjoys the advantages of imperative programming and
declarative programming. Users implement a network using the Model
API following the imperative programming style like PyTorch. Different from
PyTorch which recreates the operations in every iteration, SINGA buffers the
operations to create a computational graph implicitly (when this feature is
enabled) after the first iteration. The graph is similar to that created by
libraries using declarative programming, e.g., TensorFlow. Therefore, SINGA can
apply the memory and speed optimization techniques over the computational graph.

### ONNX

To support ONNX, SINGA implements a sonnx module, which includes:

- SingaFrontend for saving SINGA model into onnx format.
- SingaBackend for loading onnx format model into SINGA for training and inference.