GenericTensor 2.0

[WhiteBlackGoose]

GenericTensor 2.0: plan

Alright, time for improvement

1 action - 1 interface

Currently we have a large interface which forces the user to implement all its methods. It's not as good as it was thought to be:

1. You may want not to implement a method, but you can't guess whether some GT's function uses it or not
2. The interface is not extendable - adding a new method immediately breaks backward comaptibility

Instead, we will have one method in each interface (there will be multiple interfaces - like IAddAction<A, B, C> etc.).

Now we will be able to constrain each function separately, and the tensor itself won't be constrained (the type will have only one type argument - T). It makes all things much more convenient, and it also allows for more advanced API - for example, adding tensors would look like:

Tensor<C> AddElementwise<A, B, C, TAdd>(Tensor<A> a, Tensor<B> b) where TAdd : IAdd<A, B, C>

Non-generic entry point class API

Currently we need to type the typename with its arguments. Instead, there will be a non-generic static class to handle those things: thanks to type inference, we will type less.

No sparse transformation

Currently, there are some transformations, like transposition and slicing, which makes the tensor "sparse" - that is, instead of rewriting elements themselves, they rewrite the meanings of indices. That means, that going over a just created 2D tensor by width and height would be much faster than if we transpose that tensor beforehand.

In new API such a feature will remain in some different more explicit form, for example, TensorView<T>. It behaves similarly to BCL's Span<T>, which is basically a view on some piece of memory. Likewise, TensorView just allows to read/write to a tensor, but is not a tensor itself. Might be useful for some operations, where the user doesn't want to think about the order of axes or size of a tensor. Though not all API will be available for TensorView<T>.

But tensors itself will be strictly linear/dense. Slicing and transposition will produce a new dense tensor.

Hardware acceleration

Now, with dense tensors we can use three types of acceleration:

1. Multithreading
2. SIMD (HonkPerf.NET.GenericSIMD)
3. GPU

Though the third one is questioned, the first two are definitely doable and will require new methods with type constrained to unmanaged. As simple as that.

New project name - GenericTensor.NET. New type name - Tensor.

Alright, this one is questionable. Should we rename to a more "modern" name? It would mean a new "slot" in nuget packages, and hence, not visible by current users (if there are any aside from AngouriMath, hehe).

New operations

We could implement convoluted map for it. Example of convoluted map:
Imagine matrix A

1 2 3
4 5 6
7 8 9

and then code:

B = A.ConvolutedMap(width: 2, height: 2, step: 1, view: TensorView -> view[0, 0] + view[0, 1] + view[1, 0] + view[1, 1]);

then B is

12 16
24 28

This way we can basically process, reshape, "bend", collapse or add axes anyhow we need. Upscale for instance:

A =
    1 2 3
    4 5 6
    7 8 9

B = A.ConvolutedMap(
    inputWidth: 2,
    inputHeight: 2,
    step: 2,
    outputWidth: 3,
    outputHeight: 3,
    (source: TensorView, destination: TensorView) ->
        destination[0, 0] = source[0, 0]
        destination[0, 2] = source[0, 1]
        destination[2, 0] = source[1, 0]
        destination[2, 2] = source[1, 1]
        destination[0, 1] = (destination[0, 0] + destination[0, 2]) / 2
        destination[1, 0] = (destination[0, 0] + destination[2, 1]) / 2
        etc...
)

Note, that we don't limit to 2d convolution here, though not sure where we could ever need 4D+ convolutions.

Unresolved questions

Should there be single type Tensor or by dimension - Tensor1D, Tensor2D, etc.? If the latter, should there be TensorND or just cover a fixed number of dimensions?

[Darelbi]

Hi, I want to add some opinion, I'm writing Machine Learning Stuff, and soon found myself to write stuff that is not available in c#. This library is very close to what I need (and only 1 another library do that). It would save me to write that. (was halfway through)

In machine learning you can have 1D or 2D input, tough sometimes 3D is used for multiple channels of colors in images so is a thin 3D. Sometimes real 3D input is possible. And since it is more convenient to train the neural network in batches of inputs you add 1 extra dimension to all the inputs. (Basicaly pack the inputs togheter, makes the pack one dimension higher than the elements of the pack)

In a neural network there is a operation that is feed forward input which basically is

given a NDimensional series of inputs x1,x2,x3,x4 and a bias b.
and a Weight matrix/tensor (N+1 dimensional) W

the output of the layer of neural network is defined as a(W*(x1|x2|x3|x4) + (b|b|b|b)) = (y1|y2|y3|y4) .. where a is a non-linearity function like sigmoid or tanh applied element-wise to all the results.

basically the Tensor multiplication is the most common used operation and it should be heavily optimized (Parallel.For + SIMD types + dividing the loops in small cache local batches + transpose before multyplying to allow exploit SIMD Vectors better convertin a ROW by COLUMNS product to a COLUMNs by COLUMNS product). If you provide a very good tensor multiplication the library is automatically the best choice for machine learning stuff around there.

Regarding the "Number providers" if you are willing to drop support for old .NET stuff there is already the INumber interface which provides all the stuff the Number providers does in example

public static void MySum<T>(T a, T b) where T:INumber<T>
{
          return a+b+T.One;
 }

Hoping forward to see of the project, if you are willing to implement the Tensor Sum/Multiplication before all other features and release a Prerelease with just that I'll be one of your first users for sure. Also it would be nice the library support extension capability

public interface MyMultiplyBy2Operation<T>  : IUnaryOperation<T> where T:INumber
{
          public T Compute(T Input); // called on residual elements on a SIMD vector
          public Vector<T> Compute(Vector<T> input) // called on core elements to allow SIMD-ization
 }  

that would allow automatically to implement neural networks activation functions.

Regarding the types I would like just a Tensor type with size and dimensional check when you multiply or add 2 toghether.

[WhiteBlackGoose]

@Darelbi I'm glad you find the idea good. However sadly I don't have time or mood for working on this project anymore. It's an MIT-licensed project, so you're free to continue the work on your own

[Darelbi]

It's a pity. I prefer reusing existing libraries, however since this one is no longer maintained I started already doing some small changes. Thank you

https://github.com/Darelbi/NeuraSharp/tree/tensor/NeuraSharp/NeuraSharp.GenericTensor

Here it is. Right now I'm refactoring to use INumer interface, when finished I will try to look at SIMD-izing and use the fast matrix multiplication for the last 2 dimensions. I think the design is already pretty good so no need major changes.