What is Reddit's opinion of PyTorch?

From 3.5 billion Reddit comments

> 2. Use Sequential layers when possible for cleaner code.

> 3. Don't make lists of layers, they don't get registered by the nn.Module class correctly. Instead you should pass the list into a Sequential layer as an unpacked parameter.

Don't use nn.Sequential to represent a list, use nn.ModuleList. https://pytorch.org/docs/stable/generated/torch.nn.ModuleList.html

Obviously it's fine, from a code correctness standpoint, to use nn.Sequential (they're very similar data structures), but from a code legibility standpoint, you should use ModuleList, unless you're literally just stacking layers.

Here's the eager execution tutorial: https://www.tensorflow.org/guide/eager

Scroll down a bit and it shows how to create a model:

class MNISTModel(tf.keras.Model): def init(self): super(MNISTModel, self).init() self.dense1 = tf.keras.layers.Dense(units=10) self.dense2 = tf.keras.layers.Dense(units=10)

def call(self, input): """Run the model.""" result = self.dense1(input) result = self.dense2(result) result = self.dense2(result) # reuse variables from dense2 layer return result

model = MNISTModel()

In PyTorch you'd do this:

class MNISTModel(nn.Module): def init(self): super(MNISTModel, self).init() self.dense1 = nn.Linear(784, 10) self.dense2 = nn.Linear(10, 10)

def forward(self, input): """Run the model.""" result = self.dense1(input) result = self.dense2(result) result = self.dense2(result) # reuse variables from dense2 layer return result

model = MNISTModel()

It even has automatic differentiation to get gradients with GradientTape, which is equivalent to PyTorch's autograd module.

To be fair, PyTorch is adding methods to create static graphs for use in production. PyTorch and TensorFlow/Keras are converging towards the same API. PyTorch is getting there first and without the baggage of the rest of TensorFlow. If you haven't tried PyTorch yet, it is a delight to use.

I have some comments:

1. Pytorch has nn.Sequential (https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_nn.html), which is equivalent or even shorter than Keras version, you don't need to create class if you don't modified something.
2. Pytorch will be released with 1.0 version in July or August. With that version, Pytorch can work well with distributed learning and mobile device. So there will be no advantage of Keras over Pytorch in the near future.

Most things leveraging CUDA/CuDNN/CuBLAS without explicit effort to keep it deterministic. E.g. Convolution and Pooling in PyTorch on the GPU and the same in TensorFlow

> It'll be more expressive because it won't be obfuscated with unnecessary minutiae and will be developed in far less time.

When you really want to get things done in machine learning, once you're past the tutorials level, you forget Python and go straight for the C++

I work with "AI" libraries (Pytorch, scikit-learn) on a weekly basis and I can assure you it's completely overblown.

The entire field could be renamed "Predictive statistics" or something and it would be a much more apt description. But no one wants to claim their product uses "Predictive statistics". And no one is afraid of those words.

I'd love to see them if there are any. However, I think it'll likely be years before it's as easy to use / performant as CUDA / Nvidia.

Look at the difference between installing PyTorch with CUDA and installing with ROCm. IDK about you, but every hour spent messing with installs, debugging, and using someone else's Docker container just to get torch.cuda.is_available() is essentially an hour wasted.

CUDA versions can be installed on Windows, Mac, or Linux and installed with one line of Conda...

I think ultimately this is the difference between the high-level strategies of Nvidia and AMD. Nvidia has a huge AI research team, publishing in top conferences, etc. I can't turn up anything similar from AMD. Until AMD treats AI as a core business area, it won't be worth it to buy AMD GPUs.

Caveat: I'm no expert on pytorch but your question looked interesting so I read up on the documentation a bit.

Your code is not doing what you think it's doing. Pytorch multiprocessing is a wrapper round python's inbuilt multiprocessing, which spawns multiple identical processes and sends different data to each of them. The operating system then controls how those processes are assigned to your CPU cores. Nothing in your program is currently splitting data across multiple GPUs.

To use multiple GPUs, you have to explicitly tell pytorch to use different GPUs in each process. But the documentation recommends against doing it yourself with multiprocessing, and instead suggests the DistributedDataParallel function for multi-GPU operation.

You might want to read this.

With the release of PyTorch 1.0, soon, the pipeline will be researching, prototyping and training on PyTorch and deployment on Caffe2.

As of now, TF is your best bet.

> Some of the differences in MXNet when compared to PyTorch are as follows: > > + You don’t need to put the input into Variable

Not anymore, you can just use plain Tensor for everything starting with 0.4.0:

https://pytorch.org/docs/stable/autograd.html#variable-deprecated

Learning how CNNs work is the place to start. If you like PyTorch. If you like Tensorflow. The rest is really just fancy image processing.

You could get started with this PyTorch tutorial: https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

It's a mask region based convolutional neural network implementation and you can use a custom dataset as well.

PyTorch is a bit more low level than Keras but it helps a lot with understanding the mechanisms behind neural nets.

good luck.

You would implement a module that wraps all of this functionality, then use that layer to build your model.

https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules

Weights are just tensors. You can always transfer them from one framework to another. It can be a lot of effort depending on your experience and the frameworks, but usually there are libraries to help you convert a model from one format to another.

https://pytorch.org/tutorials/advanced/super\_resolution\_with\_onnxruntime.html
https://github.com/onnx/onnx-tensorflow

PyTorch 1.6 comes with Automatic Mixed Precision, but I'm not sure how it compares to Apex. Also, the zero_grad() hack looks like something PyTorch should fix on its own.

Based on what I've seen from the TensorFlow team, people I've talked to, and new tutorials, TF is moving towards an API that is similar to PyTorch. That means imperative and eager, where you can pass tensors through your network as you build it, which necessitates automatic differentiation.

This is an ease of use improvement. There is a massive cognitive jump between working in Numpy and defining static graphs in TensorFlow. The transition from Numpy to PyTorch is basically negligible because they are both strongly Pythonic and imperative APIs. In my experience, working with TensorFlow is almost like writing in a completely different language.

With a static graph, you define all the operations upfront and each of those operations has some gradient method. When you do the backwards pass to calculate gradients of your parameters, it just goes backward through the graph you defined using the gradient methods. This should also be considered automatic differentiation, just different than what PyTorch does.

PyTorch and TensorFlow's eager execution mode give you dynamic graphs. You don't define the graph ahead of time, so instead you have to keep track of the operations performed on the tensors. PyTorch's autograd and TF's GradientTape work by attaching a gradient function to each new tensor you make. For example if you have some tensor x and do y = x**2, y will be created with a method like y.grad_fn = PowBackward(), which calculates the gradient for the power operation given the output y and the input x. If all of your tensors have these gradient functions, then you can start at some tensor (the loss for example) and go backwards through all the operations leading to that tensor, calculating gradients along the way. This will eventually get you the gradients of your parameters for the SGD update step.

Start with learning Python. It's free, easy to learn, and also used in AI. If you'll have difficulties with Python, try Scratch - it's for kids learning and so even easier. Then after Scratch learn Python.

To make an AI assistant, you need some math courses (at least their fundamentals) before learning AI:

e.g. Calculus, Linear Algebra, Discrete Math, Probability & Statistics.

I don't know level of your math skills, but usually these subjects are studied in university, not in school.

After that you can study PyTorch. It's framework by Facebook for AI/Deep Learning.

There was (is?) a free textbook on pytorch.org site, called "Deep Learning with PyTorch" by Stevens, Antiga and Viehmann.

Alternative framework is Tensorflow by Google. The textbook for Tensorflow is "Deep Learning with Python" by François Chollet.

You could also enroll in Deep Learning courses by DeepLearning.AI at Coursera.

Both books and courses have projects.

I use PyTorch at work and Jax/Flax for personal projects and research, with few exceptions. I find Jax/Flax really mathematically intuitive. Still, PyTorch is generally more mature, OOP-intuitive, and easier to onboard thanks to the existing documentation, tutorials, and books (I work with interdisciplinary scientists new to DL frameworks). I'm also a huge fan of Torchscript, even though we don't use it for deployment at my company. And, it's always nice to have an "ecosystem" ready to borrow from that's built on an intuitive interface.

I haven't touched TF or Keras for a few years. I've always found it frustrating at a baseline.

I didn't see this change in the release notes, but found it on the _<u>torch_function</u>_ documentation page:

> One should be careful within _torch_function_ for subclasses to always call super()._torch_function(func, ...) instead of func directly, as was the case before version 1.7.0. Failing to do this may cause func to recurse back into __torch_function_ and therefore cause infinite recursion.

I've been playing with neural networks. It's been a fun project.

Check out pytorch. It's a fairly straight forward library that lets you build neural networks easily.

Here are some simple examples:

https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

I'm wondering when TensorFlow will get a proper, full C++ API like the one PyTorch has recently gained.

(Also, I can use CMake for PyTorch, but AFAIK am forced to use the monstrosity that is Bazel for TensorFlow...)

> prototyping and training on PyTorch and deployment on Caffe2

That's already possible via ONNX, see https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html. AFAIK 1.0 is more about making PyTorch itself deployment-ready by eliminating the hard dependency on Python and making everything natively compilable with C++.

Actually it can be dropped, but only because of the way PyTorch scales the outputs during training.

https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html

If the were not scaled during training then you'd have to scale them by p during evaluation as mentioned in the paper.

https://pytorch.org/docs/stable/generated/torch.autograd.grad.html#torch.autograd.grad

Just make a tensor with ‘requires_grad=True’, pass it through your network, then use grad to figure out the gradients like in the linked url.

If your network is completely linear with no nonlinear activations, should be only one set of gradients. Otherwise, you’ll get different gradients with different input values

I tried doing ML and i didn't have much luck with it. It is a very complicated topic, yes, but you do need to know it if you want to do anything even slightly more complex. I would start with something like text prediction (you train it on text and then it will try to predict the next word). I think I used this tutorial when learning it https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html

Have a look here for example: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html

A paper would typically just say "We applied these random augs with those parameters".

I believe your proposed approach of random augmentations at loading time is indeed valid, but much more common than you think. Have a look at e.g. transforms such as RandomCrop, RandomAffine etc. in PyTorch (https://pytorch.org/vision/stable/transforms.html), which can do exactly that and are found in many open source implementation of image classifiers.

The simplest way to do it is to load transform your images with a dataset/dataloader object. There is a nice tutorial here from the pytorch team : https://pytorch.org/tutorials/beginner/basics/data_tutorial.html

To see your augmented data, you can iterate trought your raw data and display them or save them with PIL. (Don't forget to convert your pytorch tensors to something readble by PIL, like a numpy array and to normalize the pixel value between 0 and 255).

Edit : you can also use tensorboard ( https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html)

Good work, I certainly don't have the skills to build a website like this!
But what I do find easy to use are torchvision's transforms and keras' preprocessing layers. ;)

Many thanks u/slashcom.

I googled torch.multiprocessing.spawn and found this: https://pytorch.org/docs/stable/multiprocessing.html

Almost towards the bottom, you can see torch.multiprocessing.spawn. Can I ask do you know what the two parameters for this function does: join=True and daemon=False.

Or for my piece of code, I don’t have to worry about these?

Pytorch (like the name suggests) is quite Pythonic, so if you’re not familiar with Python that’s a good start. But assuming you are, you can check out deeplizard’s course on Youtube. Make sure to use the material as a guide and do your personal readings on the topics. Moving on, you’re in luck because not long ago the official Deep learning with Pytorch book was released. All the best buddy

To see exactly where time is being spent in your PyTorch program, try out the autograd profiler. For example:

with torch.autograd.profiler.profile(use_cuda=True) as prof: <run a small number of iterations of training>

# The following will print a table which summarizes the time spent # in each operator, sorted by the time spent in CUDA kernels print(prof.key_averages().table(sort_by='cuda_time'))

# The following will emit a .json file which you can load into # chrome (navigate to chrome://tracing and press "load" and select # the json file). This allows you to see the operations plotted over # time, and also allows you to see spaces between operations where # other (Python, data loading etc) logic might be happening prof.export_chrome_trace('load_me_in_chrome.json')

Please let us know your results, as well as ways you think we might improve the profiling experience

You can read the source of the pytorch MHA module. It's heavily based on the implementation from fairseq, which is notoriously speedy.

The reason pytorch requires q, k, and v is that multihead attention can be used either in self-attention OR decoder attention. In self attention, the input vectors are all the same, and transformed using the linear layers you spoke of.

In decoder attention, the query is based on the current decoder's position, but the key and value are based on the encoder's output. But really you can have all 3 values completely decoupled -- it's just not what model's tend to do.

So reading the code you'll find these qkv_same and kv_same variables, which handle the self-attention and decoder attention cases.

So to answer your question, the pytorch MHA module does do the linear transformations like you expect. It just handles more general cases than ONLY self-attention. As far as why your code would be so much faster: it's possible, but more likely you're missing something.

You can get some pretty good results going with DCGAN which is a pretty popular image generation GAN architecture. https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html This is a link to the pytorch guide and implementation. You could perhaps try a denoising problem with it by getting some image dataset, putting artificial noise into the dataset, and see if you can reconstruct a better image? This was one of my first projects and got some neat results. Even better if you try to reimplement all the pytorch code just so you know what it all does.

Yes. See: https://pytorch.org/tutorials/advanced/cpp_export.html

Also note that there is a related but slightly different PyTorch concept called "TorchScript" for dealing with data-dependent control flow: https://pytorch.org/docs/stable/jit.html

I can't really speak to deeplearning.ai, but I think you should go with Stanford's cs231n course. I'm sure the more recent iterations are great as well, but I really like the 2016 version with Karpathy as a lecturer. Just incredibly clear and has a knack for conveying the intuition behind why things work and insights into how practitioners think.

Given that you want to learn pytorch, the assignments will be really invaluable. There are some assignments towards the end that you can do in pytorch, but the real benefit is that most of the code you'll be writing is stitching together numpy operations to create different types of deep nets, which translates incredibly well to doing the same thing with torch tensors. For getting started with pytorch, I really like this tutorial by Jeremy Howard that builds up familiarity with the different higher-level abstractions in the framework and what they are doing.

If you really want to use MAE for optimization and want better convergence, I think it's maybe better to have your own hack of the gradient at y_pred = y_true. I.e., instead of setting d MAE/d y_pred to -1 or 1 if y_pred = y_true (not sure if PyTorch uses -1 or +1), just set it to 0. (For defining your own gradients, see https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html)

It seems that pytorch has extremely straightforward implementation. https://pytorch.org/docs/stable/notes/amp_examples.html

What about starting using mixed then move to 32 later on as gradients become smaller?

Ok, so you're working with sound then. Can you detect the difference between audio clips? If not, maybe it's not detectable - too noisy input data. If yes, that's great, means the problem IS solvable.

I'd assume the first few epochs the loss is going down until the model predicts 0.333 for each class. If so, the cross-entropy should be around 1.108.

I'd try to reduce the length very much, before putting it through a dense layer - so you'd need a lot of strides.

How much data do you have? If less than, say, a thousand clips, perhaps the data is too little.

I haven't worked with audio either, but here is a tutorial.

In more detail:

• On a single machine pytorch can use Intel's &quot;MKL&quot; or OpenMP to parallelize on a single CPU.
• Going beyond that, PyTorch also has multiple options to parallelize across machines

I see you're using PyTorch. Here, they show you the pip command you need to use to set up local GPU. If you have a Nvidia card, to know which version of Cuda you need, Nvidia tells you the minimum driver version supported. For example, an RTX 3070 on Windows 10 x64 might take Stable 1.10, Windows, Pip, Python, Cuda 11.3, going through those options in order in the first link.

You might also need the Nvidia Cuda Toolkit. I can't remember. Again this is all for Nvidia cards. If you're still having trouble let me know.

Learning about algorithms is fundamental if you ever want to do something that a high-level library like pytorch or sklearn can’t do out of the box.

Also, on the deep learning/machine learning side, the Goodfellow book is fairly theoretical. There is an excellent practical book on deep learning available from some of the folks who made pytorch, which I highly recommend working through in parallel with the Goodfellow book: https://pytorch.org/assets/deep-learning/Deep-Learning-with-PyTorch.pdf

Tensor flow is a whole other thing. I have no idea where they're at.

PyTorch's info for ROCm is here: https://pytorch.org/blog/pytorch-for-amd-rocm-platform-now-available-as-python-package/ including a forum. It's relatively new but is probably worth experimenting with.

And it links to various info pages from AMD on ROCm support. I would expect ROCm to support very recent GPUs though.

I don’t think you should really need to be explicitly calculating the derivative, pytorch is going to do that for you.

https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html

https://medium.com/@ODSC/automatic-differentiation-in-pytorch-6131b4581cdf

Pytorch has a tutorial on doing some reinforcement learning that might help you a bit.

https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html

If you want to feed your network with token ids like this, you want to put an embedding matrix in front of your linear layer. You would only define a linear layer with an input dimension of the vocabulary size if you were one hot encoding the data, which you likely don't want to do.

You can read about this here:

https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html

I don't know what you mean by "ensure connections don't cascade", but there are several libraries that implement various types of neural networks. Look into PyTorch. I used it for a project once and found it intuitive to use.

You are already half answering your own question. Since you are using your own 3090 or the company quadros you are working on a single-GPU setup. When you move to deeper models (with more parameters) you will need parallel setups with several GPUs installed in different machines (a typical GPU cluster). To my understanding, knowing how to deal with DDP fulfills quite a good part of this requirement. See: https://pytorch.org/tutorials/intermediate/ddp\_tutorial.html

No, I am with you on providing the results on all the seeds (as that would help others report the statistical uncertainty in your results). But reporting only a point estimate on those seeds, which is often done to report aggregate benchmark performance across tasks, is quite bad as it ignores the uncertainty in aggregate performance.

Re determinism: non-deterministic CUDA ops can make your code stochastic even in simulation (using Jax and tf on GPU) and there is a non-trivial cost of making hardware fully deterministic: see this paper for more details.

Also, replicating someone's results often requires the same hardware too otherwise we get different results especially in RL (even with identical seeds). From Pytorch documentation : "Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds."

As you already know that this problem can easily be converted into NER problem. To do that some of the best possible approaches are CRF based token classification or most recent one BERT based token classification. (As mentioned in the other comments as well).

Do have a look at this pytorch article for CRF one, it will help you understand it better. https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html To handle your continuous words for a label, we use BIO approach while passing data to the model. (If you don't know, this article should help you).

Suggestion: If you have huge vocabulary, then build a smaller one using sub-word tokenization approaches (BPE, word piece etc) and then train CRF model to learn embedding for those subwords. (if embeddings are not readily available for your domain of data) https://huggingface.co/blog/how-to-train This article would help you regarding how to build subword tokenizer.

FYI OP, if you don't know how to use, check the example here:

https://pytorch.org/docs/stable/autograd.html#anomaly-detection

What it will do is show where in the forward calculation the error comes from.

I am not sure how to approach your question theoretically, but with my experience working with simple computer vision models, the activation functions do affect the final results because the final results are post-optimization. The optimization process requires the computation of the loss. In pytorch, one might use LogSoftmax to corporate with NLL loss when dealing with multi-class classification. After fitting, we will replace the LogSoftmax with a Softmax for interpretability.

Baselines and other RL libraries tend to be implemented with torch or tensorflow so you won’t necessarily have a lot of touching points with their inner workings.

I recommend starting with this tutorial: https://pytorch.org/tutorials/intermediate/mario_rl_tutorial.html

It goes into real detail on every (well-observed) step you listed above.

The article mentions JIT compilation with numba, but that's not the only way to compile Python code. There is also

• compiling with mypyc – it uses type annotations to convert the Python code to C and then compiles it
• compiling with TorchScript – this is in some ways similar to numba (mostly for numeric code) but, like mypyc, it needs statically typed Python code, and it can also be used for AOT compilation; additionally it can run on the GPU
• jax is again similar to numba

You're looking at the wrong place for speed improvements. Tensorflow compiles execution graph to XLA before it actually gets executed. PyTorch has introduced similar mechanisms in the last couple of releases .

If you're looking for the highest possible inference speed, you can use things like TensorRT to compile your network into something that executes much more quickly/efficiently.

In short: While all of the above languages would offer you to write in C++ instead of Python, you will not save ANY runtime by doing so.

Have a look at Feature Pyramid Networks (FPN), which use features from later layers when making predictions at earlier layers.

I'm not sure about Keras, but I know in PyTorch, there are pretrained models available to fine-tune using a variety of architectures, including RetinaNet which does object detection using an FPN.

Perhaps this is it: https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss

Specifically in the description it says:

> This is used for measuring the error of a reconstruction in for example an auto-encoder. Note that the targets t[i] should be numbers between 0 and 1.

Here, I googled it for you:

https://pytorch.org/docs/stable/tensors.html

a numpy like data structure in torch. torch implements numpy multi dimensional data structures and uses numpy's ellipses(...) notation in indexes.

It's not used by any built-in python, it's implemented a part of a package (torch) that you are using.

This might be helpful too:

https://pytorch.org/docs/stable/torch.html

Map/ Apply sounds similar to what you're looking for

If you really want one line (left out the rest of the booleans):

x = b.clone().float()
x.map_(x, lambda b, _: a[0].item() if (b < 4) else a[1].item() if (4 <= b and b < 12) else a[2].item() if (12 <= b and b < 22) else a[3].item())

I haven't tested on Linux, but theoretically it shouldn't be a problem. The batch files for installing the required packages and for running aiserver.py won't work on Linux, but you can still run the pip install commands and start the program manually from console. I'll set up a Ubuntu VM at some point and put together some shell scripts to do what the .bat files are doing.
The only other hurdle would be getting CUDA installed if you want GPU generation. The CUDA Toolkit has a Linux installer, and PyTorch has a Linux selection for their GPU-enabled package.

Have you looked at pytorch custom autograd function?

https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html

Unfortunately, the code is written in quite a coupled way so you need to pass in the optimizer and can't easily retrieve the learning rate without it.

Take the `get_lr` method from https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR and alter it to a function that takes the parameters used as arguments.

In classic ML-industry fashion, quantization is an overblown term referring to precision reduction. PyTorch docs actually have a decent TLDR of the theory and practice of quantization.

Use a vectorized env if possible, otherwise what you want to do is pass a local cpu copy of the model to each process and periodically send each process the updated state dictionary of the model. Read the official page for more info.

As you said, tricky. Interestingly, there is precedence for cv2->torch conversion within C++: https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html#implementing-the-custom-operator-in-c 💪

By default, the <code>.backward()</code> call frees the computation graph. You have to explicitly tell pytorch to keep it using the requires_grad kwarg. What the .zero() call does is zero-out the .grad tensors of every parameter being optimised.

There's named tensors in PyTorch: https://pytorch.org/docs/stable/named_tensor.html

A more comprehensive blog post here from Sasha Rush: http://nlp.seas.harvard.edu/NamedTensor2

Another option is einops/einsum type notation: http://einops.rocks/pytorch-examples.html

The results above are from the vgg-19 network, which do not contain normalization layers. (Pytorch does contain a batch-normalized vgg-19, but the results above are from the original)

Thanks for the question. The main libraries that Trankit's using are pytorch and adapter-transformers. For the GPU requirement, we have tested our toolkit on different scenarios and found that a single GPU with 4GB of memory would be enough for a comfortable use.

I think it makes sense to mention profiling your jobs as well, profiling is a great way to identify expensive/slow operations and understand where your bottlenecks are.

https://pytorch.org/tutorials/recipes/recipes/profiler.html

Here's some more info on how it's done : https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html

And here's why you'd want to do it: https://docs.nvidia.com/deeplearning/performance/dl-performance-convolutional/index.html#tensor-layout

Take a look at torchvision.transforms.functional.affine.

https://pytorch.org/docs/stable/torchvision/transforms.html#functional-transforms

Generate the random state for each input once. Call affine twice for the input and mask separately, warping each image the same way.

Make sure to use Nearest as the interpolant for your mask if it’s M-ary segmentation. Class labels can’t be averaged meaningfully in the multi label case.

Although, just clear indication of the number of dimensions and their names helps a lot. Like what PyTorch has, but in types could be nice. I've seen a library for that somewhere, but I'm having trouble finding it at the moment.

There are 3 ways to load Pytorch pretrained models. Most Pytorch models are developed using Python and saved either weights only or entire model where data is bound to Python class. You can find it here: https://pytorch.org/tutorials/beginner/saving_loading_models.html

We need to create a Python script to reload pretrained models (*.pt or *.pth), convert tensors to numpy arrays and save as *.npz. `gotch` will provide APIs to read *.npz file to Go and handle next steps of training or infering. Finally, models can be serialized and saved so that it can be reloaded directly from Go next time.

The third way is using Python script to save model to Torch Script. `gotch` will provide APIs to load JIT models.

I am personally biased towards Pytorch. I have tried using Tensorflow/Keras after learnin Pytorch and I found them very confusing/complicated compared to Pytorch. Specially the error messages in Keras/Tensorflow feels so wierd to me. Since you are new to this, you should try both and see which suits you.

You can follow this tutorial for training a CIFAR10 model in Pytorch.

https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

On another note. When applying data augmentation it is important to think carefully about the augmentations that you are applying. For example, your augmentation may randomly flip the images vertically. If you flip an image of a bag vertically it is still an image of a bag, but your test set wont have these type of bag images. So you should not use vertical flip augmentation. Only use augmentation that make sense for your test dataset.

Other than what serpye said, there is this page, by some postgrad guy who does deep learning research, who describes how to download and use his image recognition solution.

He also included a a (slightly less understandable) description of how to train the model on other data. If you're also trying to learn about image detection and deep learning, I rather recommend the introductory tutorials for deep learning APIs like tensorflow or pytorch (both tutorials are multi-page, just so you don't expect to feel done after the page I liked to specifically).

> I don't think the work we do is graphic intensive.

I was not talking about graphics. I was talking about accelerating numerical processing by using the GPU for matrix math.

Python uses the GPU to accelerate machine learning algorithms and for matrix math.

So can Matlab for accelerating some of its math math.

The workflow is basically:

1. Set up a relational database for logging documents
2. When training a classifier, stream data from the database to the training nodes

My company mostly uses some internal tools for this, but the best thing would probably be to migrate all your data and training to AWS. I know you can reserve HIPAA-compliant RDS instances. I've never used Amazon Sagemaker, but it seems well-suited for this. Basically as the volume grows, you want to be able to stream your data - e.g. in PyTorch, using DistributedDataParallel. Your EC2 instance can just stream data from RDS at training time. I've never tried them personally, but gensim has distributed versions of LSA, LDA and some other clustering and vectorization algorithms. And you can probably get a lot of insights from doing simple text queries.

Semi-relevant, when my fiancee was working on precision medicine stuff, her PI wasted so much time and money on GCE VMs with huge amounts of RAM to run RStudio because of some random generics libraries. It's a lot better to just bite the bullet up front and try to architect your data and code to be distributed, even if it takes some engineering time to figure out. Especially since it looks like precision medicine is moving that direction anyway.

You can make an iterative dataset class to handle the IO. Your dataset class will work like an iterator and read files line by line. It can be used together with dataloader as well.

Example: https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

You have used the wrong functionname to define the length of the dataset, you are using __length__ while you should use __len__ as per the pytorch docs.

Thank you!

I'm happy I finally got it running. The dependencies were all screwed on my end. Don't know what's going on with my visual studio installation for instance.

If anyone runs into similar issues. Just install the missing packages with 'pip install' and install the correct pytorch by using the command line generated on this site: https://pytorch.org/get-started/locally/

If your vs install is crapped like mine, just only install the binary packages, for instance: "pip install --only-binary :all: bimpy"

Edit: I'm running it on a 970 and it runs at an estimated 6 fps or so.

Edit2: Smile>15 gets nuts, Smile <-30 loses the left eye for some reason..

https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html

I thought the tutorials on their website were fairly straightforward! I had never done anything neural net beyond a couple Keras examples and I was able to follow along pretty easily. I have a strong programming background already but honestly it’s almost an entirely different skill beyond basic syntax and procedural thinking.

It honestly depends on what you're struggling with. If it's the actual ML then it might be better to focus on learning that. If it's something library specific I think it's better to step through the code and figure it out that way. PyTorch has some pretty good documentation, just look up everything that seems odd.

PyTorch code is typically broken up into 3 or so pieces. Managing the data, defining the network and running/using the network. Data management is typically what changes the most from project to project.

I would consider what you linked to be a more advanced example of using PyTorch. If this seems difficult it might be better to check out their training a classifier example.

I'm not familiar with the implementationd details, but convolutions usually use zero-padding. If you use circular padding instead, your network can learn tileable features.

In pytorch for example, you can supply padding_mode = 'circular' to Conv2D.

The course uses fastai library to simplify data processing and data cleaning. For building neural networks and training them pytorch is used. matplotlib is used for plotting graphs. Whenever they use a piece of code from fastai library, the under lying logic is clearly explained or a resource where one can learn in depth details of implementation is provided.

I personally never used fastai library at work but I still found the insights I gained from the course very valuable.

I am not familiar with the data-camp course, thus I cannot comment about it.

As you mentioned, fastai is the most time effective way to learn the concepts. I could have easily spent hours figuring out some of the explanations provided in the course, and I seldom found a better explanation in other places.

Hope this helps :)

Sorry I wasn't being very clear... I meant actually coding GPU kernels by leveraging Julia's JIT compiler as in e.g. here: https://mikeinnes.github.io/2017/08/24/cudanative.html

&#x200B;

It would be really great to write whatever RNN architecture or weird convolutional layer as a for loop and just have that compiled into a fast CUDA kernel.

&#x200B;

PyTorch's torchscript tries to do something like that (this is a very nice post) , but that's basically just another hack on top of Python I guess...

See the weight parameter of the NLLLoss definition here: https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss.
You can find some high-level examples of how per-class loss weighting can be used at https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras.

layer = nn.Linear(41* 2048, 2048)

then

layer(input_tensor.reshape(-1, 41 * 2048))

or something similar

torch.Tensor.reshape()

It is not. In fact the Variable API has been deprecated.

If you look at the source code of the RNN class itself, you'll see that if an initial hidden state is not passed in, it simply creates a torch.zeros().

Haven't read the blog post yet, but I can give you an example from personal experience.

I trained a VAE which included some max_(un)pooling layers.

I stored the indices from the pooling operation and used them again for the unpooling, due to the ambiguous output size of the unpooling layer (https://pytorch.org/docs/stable/_modules/torch/nn/modules/pooling.html)

I noticed something was wrong when I started traversing/manipulating the latent code (=the output of the encoder) before giving it to the decoder. The output of the decoder seemed unaffected from changes in the latent code. On the other hand, even slight changes in the pooling indices destroyed the output completely. I absolutely didn't expect that, but apparently it's a thing. And using constitutional layers instead of pooling solved the problem.

I don't see the necessity to put that in a video that simply shows code in an interactive python shell without code highlighting and without a demonstration on how to use the module.

This is covered way shorter in the official PyTorch custom nn.Module example.

Good narrator voice though!

Hi, I read your paper and Pytorch implementation. I have a couple of questions.

When you implemented teacher forcing algorithm, you CONCATENATED previously generated token with the input token. This is different from [what I know](https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html), where ONLY previously generated token is fed as an input. I guess it's a solution you came up with to deal with the situation where previously generated token and input token are in different language? Could you provide a justification for this apporach or other references that resemble the way you do?

You used BPE tokenized sentences where as the baseline model(Bahdanau et al., 2014) did not use BPE. I think you should try your model without BPE to make a fair comparison. What do you think?

Thanks in advance!

In one of the next releases this summer (PyTorch 1.0), python packages for Caffe2 and PyTorch will be merged into a single package in pip and conda, providing both of their functionalities. It focuses on improving scalability and performance. So you could wait for that release. https://pytorch.org/2018/05/02/road-to-1.0.html

I will try briefly explain the exception message:

This exception is raised when the Base class to create custom autograd.Function is directly called. Link below explains how to do it correctly

Please check simple steps to reproduce the exception and fix for them here: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)

Well with Pytorch you can just check your model to see whether the layers you're training are float32 or half precision (float16), that's easy.

Then there are a number of flags to set within Pytorch that could have an impact on your CUDA performance / behavior. I would check the available cuDNN flags (https://pytorch.org/docs/stable/backends.html?highlight=cudnn#torch.backends.cudnn.benchmark), as these can effect performance and behavior of the CUDA backend. I would also look at the Pytorch Forum, a number of members of the Pytorch dev team are active there and are extremely helpful.

It depends on how you're storing your data. If its some sort of vector type data made up of just numbers, you can try storing your data in the HDFS format.

This format lets you randomly access the file at any index at quite high speed (this will still be a bit slow if for example, you're using a HDD to store your data).

Another method is to use an IterableDataset. In this, your dataset has an '__iter__' function. You can split your dataset into say 10 * 5Gb, and implement your dataset such that it loads 1 file, then iterates through it returning the required data, then once you've reached the end of the file load the next one.

Why would you need or want a ML model to create noise - just use the building blocks that the framework uses to generate pseudorandom numbers:

• https://pytorch.org/docs/stable/generated/torch.rand.html
• https://www.tensorflow.org/api_docs/python/tf/random/uniform

Looking at your two code snippets, it looks like your init function is the same, causing the network to expect a different set of inputs than what you're providing. You can use Pytorch's inbuilt RNN modules as shown [https://pytorch.org/docs/stable/generated/torch.nn.RNN.html](here)

torch datasets and data loaders are made for this type of thing:

https://pytorch.org/tutorials/beginner/basics/data_tutorial.html

Besides making everything easy, the big benefit is that data loaders will load extra batches into memory while the GPU is doing its work, so your network won't have to wait for data.

I'm guessing you're doing something like a denoising autoencoder? I don't do any vision stuff, but you can use data loaders to add noise to images at batch generation time. I'm 99% sure this will work better than only having one modification per image.

Ah ok, that explains that.

Yeah it's possible to run it locally, but will take some time to set up. Also you'll need a beefy GPU. And honestly will take some programming knowledge because it's not all straightforward.

You'll need to install python and then pytorch, this should help with that: https://pytorch.org/get-started/locally/ Then run al the python code from the notebook locally. You'll also need to change some location paths for the model and the output. When running the code there will probably be some packages that are not installed, but you can use pip install <packagename> to install those in the command line.

> I haven't the slightest idea about any of this.

Then why are you being so defeatist? I thought you were responding from a position of experience and had some reason to suspect that ONNX wouldn't work here.

ONNX is a format, not a language. "Converting" a model to ONNX with pytorch just involves running a forward pass through the model and then invoking an export method.

https://pytorch.org/docs/stable/onnx.html

I haven't actually performed the procedure myself, but that doesn't mean I "know nothing about it." Maybe you should actually click on some of those links I sent you. Crazy idea.

You wanna give up before even trying, you do you. There are a lot of options available for you to pursue, or you could just pretend like you don't have a GPU at all even though we live in a world where these models can be run on your CELLPHONE because of tools like the ones I directed you to.

Maybe save the snark until you've actually tried something first. You're welcome.