What are /r/pytorch's favorite Products & Services?

From 3.5 billion Reddit comments

You would implement a module that wraps all of this functionality, then use that layer to build your model.

https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules

https://pytorch.org/docs/stable/generated/torch.autograd.grad.html#torch.autograd.grad

Just make a tensor with ‘requires_grad=True’, pass it through your network, then use grad to figure out the gradients like in the linked url.

If your network is completely linear with no nonlinear activations, should be only one set of gradients. Otherwise, you’ll get different gradients with different input values

The simplest way to do it is to load transform your images with a dataset/dataloader object. There is a nice tutorial here from the pytorch team : https://pytorch.org/tutorials/beginner/basics/data_tutorial.html

To see your augmented data, you can iterate trought your raw data and display them or save them with PIL. (Don't forget to convert your pytorch tensors to something readble by PIL, like a numpy array and to normalize the pixel value between 0 and 255).

Edit : you can also use tensorboard ( https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html)

Many thanks u/slashcom.

I googled torch.multiprocessing.spawn and found this: https://pytorch.org/docs/stable/multiprocessing.html

Almost towards the bottom, you can see torch.multiprocessing.spawn. Can I ask do you know what the two parameters for this function does: join=True and daemon=False.

Or for my piece of code, I don’t have to worry about these?

You can read the source of the pytorch MHA module. It's heavily based on the implementation from fairseq, which is notoriously speedy.

The reason pytorch requires q, k, and v is that multihead attention can be used either in self-attention OR decoder attention. In self attention, the input vectors are all the same, and transformed using the linear layers you spoke of.

In decoder attention, the query is based on the current decoder's position, but the key and value are based on the encoder's output. But really you can have all 3 values completely decoupled -- it's just not what model's tend to do.

So reading the code you'll find these qkv_same and kv_same variables, which handle the self-attention and decoder attention cases.

So to answer your question, the pytorch MHA module does do the linear transformations like you expect. It just handles more general cases than ONLY self-attention. As far as why your code would be so much faster: it's possible, but more likely you're missing something.

This is specific to NLP, but it should be roughly equivalent to the material they cover in the stanford course for NLP so it should get you to a pretty advanced stage... https://smile.amazon.co.uk/dp/1491978236/ref=cm_sw_r_cp_apa_fabc_2--bGbSHBY492

If you want to feed your network with token ids like this, you want to put an embedding matrix in front of your linear layer. You would only define a linear layer with an input dimension of the vocabulary size if you were one hot encoding the data, which you likely don't want to do.

You can read about this here:

https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html

Map/ Apply sounds similar to what you're looking for

If you really want one line (left out the rest of the booleans):

x = b.clone().float()
x.map_(x, lambda b, _: a[0].item() if (b < 4) else a[1].item() if (4 <= b and b < 12) else a[2].item() if (12 <= b and b < 22) else a[3].item())

Unfortunately, the code is written in quite a coupled way so you need to pass in the optimizer and can't easily retrieve the learning rate without it.

Take the `get_lr` method from https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#CyclicLR and alter it to a function that takes the parameters used as arguments.

By default, the <code>.backward()</code> call frees the computation graph. You have to explicitly tell pytorch to keep it using the requires_grad kwarg. What the .zero() call does is zero-out the .grad tensors of every parameter being optimised.

The results above are from the vgg-19 network, which do not contain normalization layers. (Pytorch does contain a batch-normalized vgg-19, but the results above are from the original)

Take a look at torchvision.transforms.functional.affine.

https://pytorch.org/docs/stable/torchvision/transforms.html#functional-transforms

Generate the random state for each input once. Call affine twice for the input and mask separately, warping each image the same way.

Make sure to use Nearest as the interpolant for your mask if it’s M-ary segmentation. Class labels can’t be averaged meaningfully in the multi label case.

You can make an iterative dataset class to handle the IO. Your dataset class will work like an iterator and read files line by line. It can be used together with dataloader as well.

Example: https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

It is not. In fact the Variable API has been deprecated.

If you look at the source code of the RNN class itself, you'll see that if an initial hidden state is not passed in, it simply creates a torch.zeros().

I don't see the necessity to put that in a video that simply shows code in an interactive python shell without code highlighting and without a demonstration on how to use the module.

This is covered way shorter in the official PyTorch custom nn.Module example.

Good narrator voice though!

No.

Source: I develop my own deep learning framework for a little-known language called Nim and I've been deep diving into the source code of Numpy, Scikit-Learn, Pytorch, Neon, Theano, Chainer, Tensorflow and Mxnet a lot before making design decisions. (https://github.com/mratsim/Arraymancer)

mastering pytorch? this one? never heard of that

https://www.amazon.com/Mastering-PyTorch-powerful-architectures-advanced/dp/1789614384

there a new book by Raschka about pytorch, surely it's good

I will try briefly explain the exception message:

This exception is raised when the Base class to create custom autograd.Function is directly called. Link below explains how to do it correctly

Please check simple steps to reproduce the exception and fix for them here: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)

It depends on how you're storing your data. If its some sort of vector type data made up of just numbers, you can try storing your data in the HDFS format.

This format lets you randomly access the file at any index at quite high speed (this will still be a bit slow if for example, you're using a HDD to store your data).

Another method is to use an IterableDataset. In this, your dataset has an '__iter__' function. You can split your dataset into say 10 * 5Gb, and implement your dataset such that it loads 1 file, then iterates through it returning the required data, then once you've reached the end of the file load the next one.

loss.item() is causing a blocking synchronization and copy from GPU to CPU. You can keep running_loss on the GPU and only sync it to CPU ever K batches instead.

See https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html#avoid-unnecessary-cpu-gpu-synchronization

Check out torchio for loading and processing your 3D data. And conv3d architectures in pytorch for your model. If its a segmentation problem checkout unet

not expert of gpu, but i think that you need to look for the complexity of "coalescing" duplicates on the GPU. If vector is purely sparse (without dense support), summing values along duplicated coordinates requires sorting or something like a hash table with atomic operations (i am not sure but i think hash tables on gpu is an active area of research...)

https://pytorch.org/docs/stable/sparse.html#sparse-uncoalesced-coo-docs

The first issue I see here is that your model is overfitting (train loss <<< validation loss). The second issue is that your dataset is imbalanced. You can either shuffle the dataset in the dataloader or just regularize it and perhaps drop some outliers.

Categorical Cross Entropy Loss in pytorch is a combination of Log Softmax and Negative Log Likelihood loss, which are different than adding a Softmax on your own. https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

Also, you're using only Linear - an affine transformation mimicking a linear regression instead of logistic regression (where you fit a logistic function on your data). Currently what you're doing is

y' = Wx + b

loss = ce_loss(y, y')

What you should be doing is:

y' = Wx + b
y' = softmax(y') # or you can use sigmoid for 2 classes

loss = ce_loss(y, y')

Try torch.maximum rather than torch.max, as it computes the elementwise maximum. The goal here is to compare each element in the array to zero and not to find the max value of the array: https://pytorch.org/docs/stable/generated/torch.maximum.html

you can easily calculate based on the parameters by checking https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html . Scroll down to shape and manually calculate.

What you’re looking for is a custom data loader

https://pytorch.org/tutorials/recipes/recipes/custom_dataset_transforms_loader.html

Look this up you’ll get an idea in case you have any questions feel free to dm me.

I saw in another comment you’re using b as a Boolean mask. In this case, you can use normal multiplication and broadcasting, you just need to add the extra dimensions to b so that it’s broadcastable with matrix a.

All you’ve gotta do is add some singleton dimensions. Just use torch.unsqueeze twice on matrix b to add a third and fourth dimension (https://pytorch.org/docs/stable/generated/torch.unsqueeze.html) and multiply as normal

How do you want to multiply them? torch.sparse.mm works but you’ll need to slice tensor a and recombine depending on how you want the multiplication to work.

https://pytorch.org/docs/stable/generated/torch.sparse.mm.html#torch.sparse.mm

Did you use tensorflow?

Edit: sorry I was forward here from a different subreddit. I see now it is pytorch.

Edit2: here is a helpful resource: https://pytorch.org/tutorials/beginner/saving_loading_models.html

There is register_forward_pre_hook and register_forward_hook ( https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html#torch.nn.Flatten.register_forward_hook). The pre hook is called immediately before forward, the regular one is called immediately after forward. If you want to modify the inputs to the layer use pre, to modify the outputs use the the other.

The hooks can be assigned to any module, so they can be registered to the individual layers, blocks, or entire model. For example, say your model has 3 separate sequential models, each one with 2 layers. You could register forward hooks for specific layers in each sequential (modify each layer output), on the sequential (modify the output of the block), or to the entire model. There is an example of how this works at the end of this video: https://youtu.be/1ZbLA7ofasY

This blog post covers some basic things to look for. There was also a previous post here with some other methods for speeding up model training.

self.a = torch.nn.Parameter(torch.rand(5))

https://pytorch.org/docs/stable/generated/torch.rand.html

https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html?highlight=parameter

The backward function already computes the gradients. After running the code above, what gives nx.fc1.weight.grad? There are tutorials who can explain much better than me, checkout: https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html

What're you hoping the returned values will be here and how are you hoping this will work? (i.e., what are you hoping this will tell you, and how?)

The error itself is pretty straightforward... all doesn't support type torch.cuda.FloatTensor, but requires either type bool or type uint8. Since your input is not of either of these two support types, you're getting the error.

&#x200B;

Perhaps this is closer to what you're looking for?:

torch.eq(a, b) (in doc)

&#x200B;

Which you might use like this:

assert (not torch.all(filled_target[:,-forecast_length:,:].eq(trg[:,d-forecast_length:decoder_seq_len,:])))

u/PitifulWalk354, Your nvidia-driver version 396 corresponds to CUDA 9.2 (you can cross-check from this link). So, you need to install a pytorch version which has been compiled for CUDA 9.2 or lower. A quick look from pytorch official site shows that v1.7.1 and below support CUDA 9.2.

I think in your case the issue could be that the requirement file downloaded/installed a torch version that has been built for CUDA 10 or above (I think you are using anaconda, so check the result for conda list and check which version of cudatoolkit is installed, pretty sure it would be above version 9.2). If that is the case, simply remove pytorch (since you used requirements.txt to install it, so you can remove it by pip uninstall torch) and install the correct version using the command conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=9.2 -c pytorch

If you run nvidia-smi it will tell you your CUDA version. Then go to https://pytorch.org/, scroll down to "INSTALL PYTORCH" and select that CUDA version and your python version, environment etc. Follow the installation instructions.

Check out convtranspose2d, its the closest to a deconvolution operator and most papers use this when they say deconv.

https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose2d.html

Ah, you're getting error because there is extra .module in the keys, Yes? Three options: Wrap your model in dataparallel before loading weights Modify each key in checkpoint dict Save checkpoint again, make sure to bring the model to cpu before saving.

Read up on pytorch tutorial on saving weights: https://pytorch.org/tutorials/beginner/saving_loading_models.html

cross entropy loss does not want one-hot vectors but rather direct numbers. see https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html paragraph "Shape"

characters = torch.max(Y, 1)[1] loss = self.criterion(out, characters)

In case you actually wanted to compute entropy with a vector of probabilities (e.g. because it's not only one-hot encoding, but also with soft labels), you can use KLDivLoss

Thanks. Yeah I managed to get checkpointing to work by saving the optimizer and loading it via its state dict (optimizer.load_state_dict)

Basically just followed this:

https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html

You need to go here: https://pytorch.org/get-started/locally/ and enter your setup (windows vs Mac etc). Then it will give you a line to copy that says “pip install ...” you have to use regular pip on the command line, Pycharm package management usually doesn’t work for me with pytorch.

You could implement it by hand coding the backward pass Eg.: https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html

OFC, you’ll need to use the derivative of the tanh in the backward method, not the tanh itself.

If you google straight through estimator you’ll find implementations similar to the one you want (though using a max on forward and grad. Softmax on backward)

In order for broadcasting (torch docs / numpy docs) to yield correct results. If you didn't have a new axis, the shapes would be incompatible and torch.max() or torch.min() would fail due to a mismatch.

Adding a new axis on the 2nd dimension makes it broadcast the operation on that dimension (that's called padding)

I'm guessing they're just helping create the windows version for PyTorch and other libraries like TorchVision.

https://pytorch.org/blog/microsoft-becomes-maintainer-of-the-windows-version-of-pytorch/

Okay, I would inspect the images coming in from your dataset, make sure they're in the range of 0.->1.

Maybe you could use a dataset class like the one used here to properly batch your stuff, should make things easier in terms of less pitfalls

https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html

&#x200B;

I always start out with a toy dataset when making something, so that I know that my arch works, so maybe try mnist, then you can develop your own dataloader knowing that your training loop and architecture works.

Found official introduction about that:
https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html

Now It became more clear: * trace is general approach: you save computation graph when execute forward * script is used when you need conditions and loops

you can look in the original inception paper for the details, but for pytorch there is an implementation in torchvision:

https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomResizedCrop

in practice I have found that it can make a huge difference, always try it!

something like,

transforms.Compose([ transforms.Resize((256)), transforms.RandomResizedCrop(scale=(0.16, 1), size=227), transforms.RandomHorizontalFlip(), transforms.ToTensor(), transforms.Normalize(mean=[104 / 255.0, 117 / 255.0, 128 / 255.0], std= [1.0/255, 1.0/255, 1.0/255]), ])

Hey!

- The `set` command only sets the environment variable for the current terminal, and `setx` sets it globally.

- You might need to include Python if you want to develop custom C++/CUDA extensions, as explained in this tutorial. Including Python is optional, feel free to skip it if you don't need it.

- Yes you can load your Python trained models in C++. You can follow the steps in this tutorial to do that.

Those are models, probably pytorch models to be exact.

Here’s how to load them into your program. Pytorch docs are pretty solid.

https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model

Pytorch uses automatic differentiation. Basically, it tracks all the operations and computes the gradients using the chain rule. Look at pytorch’s docs on autograd for more details.

X = torch.Tensor([1,2,3,4]) Y = torch.Tensor([4,3,2,1,0,1])

fc1 = nn.ModuleList([nn.Linear(4, 2) for _ in range(3)])

temp = [] for l in fc1: temp.append(F.relu(l(X)))

xt = torch.cat(temp) print(xt) print(F.mse_loss(xt, Y))

returns:

tensor([0.0000, 0.0000, 2.1010, 0.0000, 0.0000, 0.0000], grad_fn=<CatBackward>) tensor(4.5017, grad_fn=<MseLossBackward>)

torch.cat() documentation here: https://pytorch.org/docs/master/generated/torch.cat.html

Hope this helps !

Setup an environment to run PyTorch, it is extremely easy to run it on Cloud with Colab or Locally on Conda.

Run the beginner example then print/debug any variable you are not sure from top to bottom https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html

If you just want to run it using pytorch and don't care about the code itself, you could use ONNX to port the model over (depending on support for your model's components).

• https://github.com/onnx/tensorflow-onnx
• https://pytorch.org/docs/stable/onnx.html

Thank you, you have been really helpful.

I see this makes sense to me and I will try it out and see how it works. I am still a bit confused on exactly what EmbeddingBag is doing and wanted to check my understanding.

My understanding is that it is taking my tfidf feature matrix which is N x V (N: num_docs, V: vocab_size) and will put it into some latent space where I can specify the dimension of the latent space (i.e embedding dimension). Typically the embedding bag weights everything equally but I can pass the tfidf values and it will account for that. The mode parameter which specifies the way to reduce the bag pretty much just gives me choice of how output is created of the "embedding layer" which I pass to my next layer.

I guess one of my goals (early in research stage) is to just do a create a baseline Logistic Regression in Pytorch which is as fast as SciKit Learn. But I don't think the embedding bag will work for just plain logistic regression (am I mistaken)?

I have modified my dataset class to have 3 dense vectors (col_id, data, indptr). Do you just have a collate function which combines these values into a batch. If you have a small code sample that would be extremely helpful?

Really appreciate it

https://pytorch.org/tutorials/beginner/nn_tutorial.html, look at this, this explains how you can initialise your own weights and bias, and do updates to them based on error, you can try to modify the parts after the updates to implement your own custom rounding off.

Just check out the pytorch docs. There's a ton of stuff focused on NLP, and it's all guaranteed to be current.

https://pytorch.org/tutorials/#text

Another good source: https://allennlp.org/tutorials

Switching from Keras to Pytorch is simpler than you might think. I really recommend the 60 minutes tutorial. From there you can reimplement things you have done in Keras in Pytorch. Furthermore I would recommend Github, find some project where you understand the theory and see how other people implemented it.

pickling the model will only work if you ship the code with the model and don't make any changes to the layout of the files between the time when you save it and when you load it because when you unpickle it it will try to import the class and instantiate it.

If you can jit compile your model than you should be able to save it with https://pytorch.org/docs/stable/jit.html#torch.jit.save, which includes the architecture definition.

Sorry if original post was too vague. Long time pytorch user but in research setting. Now trying to use it at scale in production and looking for a solid training and deployment story from actually users before I professionally committing to technology. Distributed training and model serving documentation seems descriptive thinking of going with microservices architecture as described here https://pytorch.org/blog/model-serving-in-pyorch/. Has anyone here used this in production?

Load the pretrained model. For the frozen weights, set requires grad to False. Then, update the architecture of the last layer(a) as needed.

There’s an example of this under the “fine tuning” section here: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

It depends how you create the tensor.

If you use torch.rand(), torch.zeros(), etc., you'll get a 32-bit float - that's a dtype of torch.float32 and a tensor object of type torch.FloatTensor (or torch.cuda.FloatTensor on GPU).

If you initialize with data, it depends on the data. If all the cells of your tensor are integers, as in your example above, you'll get a tensor with a dtype of torch.int64 and class tensor.LongTensor. If, instead, you had initialized it with floating-point data - e.g., torch.tensor([2., 3., 5]) - you'll get 32-bit float again.

Any of these can be overridden when you create the tensor. For example, if you create a tensor with torch.tensor([1,2,3], dtype=torch.float64), your integer values will be cast to floats and you'll get a tensor with the object type and dtype you requested.

You can also set your default type globally within a run context. As @SirRantcelot points out, you can use torch.get_default_dtype() to find your current default type (the one that you get with torch.rand() etc.), but you can also set it. For example, you can set all your new tensors of unspecified type to be 64-bit floats with torch. set_default_dtype(torch.float64). (See docs for more info on this.)

If you want to change the dtype of a tensor, you have to make a copy. For example:

>>> t1 = torch.tensor([1,2,3])
>>> t1.dtype
torch.int64
>>> t2 = t1.type(torch.int16)
>>> t1.dtype
torch.int64
>>> t2.dtype
torch.int16
>>> t2
tensor([1, 2, 3], dtype=torch.int16)

Hope this helps!

I don't know how they are initialized by default. I would experiment with self.ln_1.{gain,bias}.data. Check LayerNorm.__init__ here.

https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv2d If you look at the source you'll see that the Conv2d module calls F.conv2d in its forward function. There shouldn't be any significant performance difference.

However, the initialization might be different. It looks like you are initializing the weights sampling from a uniform random where as the Conv2d module uses He Initialization (I think. Might need to look into the source to confirm that. It is definitely something a little more fancy then just random). I don't have good sense of exactly how much this would matter, but it might affect slightly convergence. In general unless you are doing something really fancy, you might as well use the built in Conv2d module.

Your question is kind of weird, but what you could do is download the corresponding .whl file from the website and simply install them to any computer using pip install <path to the .whl file>. Just make sure you download the .whl according to your specifications.

Have you looked at any of the tutorials hosted along with the documentation? There are quite a few. I think you'll find this one helpful.

Oh, so GANs are two networks combined into one, if there is training scripts in this repository it might show you how to create the combined model that you'll need to trace.

https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html

^ There's also this very good tutorial on the PyTorch website that might be able to help :)

https://pytorch.org/docs/stable/notes/autograd.html

> If there’s a single input to an operation that requires gradient, its output will also require gradient. Conversely, only if all inputs don’t require gradient, the output also won’t require it. Backward computation is never performed in the subgraphs, where all Tensors didn’t require gradients.

PyTorch is able to easily read in data structured in that format (where images are split into folders where the folders have the class names) using the ImageFolder function. I'd check out the Transfer Learning tutorial which goes over an example that uses the ImageFolder function to load in data: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

You can implement the same functionality using your own custom Dataset class if you want to go that route.

I don’t really know what causes the problem, but maybe try using/installing anaconda (https://www.anaconda.com) to create a virtual conda environment conda create -n my-conda-env python=3.6, activate the environment and then use conda install torch torchvision to install the packages.

Let me know if it helped (it’s mainly just a blind shot).