What are /r/deeplearning's favorite Products & Services?

From 3.5 billion Reddit comments

I haven't did the math, but this

https://www.paperspace.com/pricing seems to be cheaper

I thing in OVH you can find cheaper ones

Google cloud charges you per training instead of per hour, so it could turn out to be cheaper depending on the project.

There are a lot of alternatives, and probably some of them could be cheaper if you don't mind to lose all the other things AWS offer you.

Anyway the difference is not huge, look at what suits you better.

I think a good keyword to search for is 'multi-modal deep learning', for example, there are multiple papers on Multi-Modal Variational Autoencoder that might be a good read (e.g. this one). Basically, they try to encode data of two or more modalities (molecules and music) into a shared latent space so that the model tries to find common patterns in the data and reconstruct the different input data structures with independent decoders.

It seems that pytorch has extremely straightforward implementation. https://pytorch.org/docs/stable/notes/amp_examples.html

What about starting using mixed then move to 32 later on as gradients become smaller?

Sure. Here's the link to the 1st article : https://medium.com/@julsimon/an-introduction-to-the-mxnet-api-part-1-848febdcf8ab

Regarding pricing, you'll find everything here: https://aws.amazon.com/ec2/pricing/on-demand/

The most economical GPU instance is g2.2xlarge ($0.65 / hour, us-east-1). It has a single GPU which should be enough for a small dataset. Hard to give exact numbers, but a few $ per training sound about right :)

Yeah, let's! Taking the photographs might not even be necessary since there are already photo databases (Pixabay for example) with public domain photos.

I would think that the best way to go about it would be to create a web interface that shows people photos and asks them to label/annotate them. This way the barrier to start contributing would be extremely low.

Tesseract OCR is a solid OCR tool. Looking at how it's built may inform your approach, and you may even be able to feed Tesseract's network outputs into your own model to improve it.

These two are good. First one has good introductory explanations. Even available as audiobook, which can help reinforce concepts.

Deep Learning with Python https://www.amazon.com/dp/1617294438/ref=cm_sw_r_cp_api_i_vf0pFb3Q4WGNC

Deep Learning with PyTorch https://www.amazon.com/dp/1617295264/ref=cm_sw_r_cp_api_i_ge0pFb2ZTB6PE

Gotcha. Mining cases are not a good option.

Open frames like this https://www.amazon.com/Mining-Computer-Currency-Bitcoin-Accessories/dp/B09CNG58R1/ should work. Building your own base on available space is best though.

oh, I wasn't aware there was a new version.

The one here (https://www.udacity.com/course/deep-learning-nanodegree--nd101) still has the same syllabus as the freeCourses one though.

What's the new material you want to learn that's not covered?

https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893 also uses pytorch but is pretty good. Does a good job of teaching the fundamental and practical aspects of deep reinforcement learning. A sample of the projects you will build https://github.com/karanjude/DeepRL

>As for quad 3090 setup I recommend a rig-like setup with enough spacing between cards and pci-e gen 4 riser cables.

Thanks! Any recommendations for 4-GPU rigs?

For example, I found this mining rig. Does this work? Thanks.

https://www.amazon.com/MLLIQUEA-Machine-Platform-Barebone-Motherboard/dp/B096ZP24X4

One thing I discovered by accident at an earlier point in this process was how much a song is determined by the color of the sounds used and/or instruments selected.

I think Facebook and Google have both experimented with this, but I can only find information on Facebook for now: [https://thenextweb.com/artificial-intelligence/2018/05/22/facebook-made-an-ai-that-convincingly-turns-one-style-of-music-into-another/

How I understand this task is that the first part would be to segment the foreground object(s) in the image. The second part would be to get the 'floor' area of the elevator/parking lot and then integrate the area around the foreground object(s) to get a free-space estimate.
For the first part, my first intuition on that is to look for models that perform foreground-background separation of the image. There are multiple examples of that in classification models and also segmentation models (something like this, or this) I googled 'CNN foreground background' and found multiple papers on that.

The second part might be generally solvable by doing edge detection and search for plausible arrangement of straight edges that identify e.g. a parking lot. However, for an exact computation of the area in meters, the camera distance/orientation/scaling should be known, right? For more specific ideas, more a more precise description would help.

First try switching on shuffling on the dataloader. That's the easiest solution, and might be enough. It doesn't guarantee that all GPUs will always get different data, but you're mainly concerned with making sure the GPUs aren't all getting entirely the same batches. If you're still seeing the model diverge, it might be worth adding in a sampler that handles multiple GPUs. Some examples are available here.

There's Horovod, but that requires building PyTorch with MPI support. Easier way is to use torch distributed, which is already build into PyTorch. You can find instructions here, but basically you need to add

> torch.distributed.init_process_group()

At the beginning of your training script. Then wrap your model and optimizer with ddp

> local_rank = int(os.environ['LOCAL_RANK']) > model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank], output_device=local_rank)

then launch training with

> python -m torch.distributed.launch --nproc_per_node=NUM_GPUS_YOU_HAVE YOUR_TRAINING_SCRIPT.py

This will set the local rank env variables, and launch training on each node. Also, make sure to shard or randomize your data across the different gpus so they're not all replicating the same data.

Honestly, I'm not the best person to give tips there because I would rely on other people / high level classes when doing C++ work.
If I was to try deploy a model in C++ now, I would use something like the PyTorch C++ API or Torchscript
https://pytorch.org/cppdocs/

You can have a look at style transfer or the recent advances in GANs (VQGAN for example). The approach you would want to try is called fine-tuning, meaning retraining an already pretraines model on your data. In terms of programming framework I guess pytorch would be the best pick. Style transfer: https://pytorch.org/tutorials/advanced/neural_style_tutorial.html VQGAN: https://github.com/CompVis/taming-transformers

Pytorch docs explicitly reference his videos too lol

"Backward Propagation: In backprop, the NN adjusts its parameters proportionate to the error in its guess. It does this by traversing backwards from the output, collecting the derivatives of the error with respect to the parameters of the functions (gradients), and optimizing the parameters using gradient descent. For a more detailed walkthrough of backprop, check out this video from 3Blue1Brown."

https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html?highlight=parameter

Sadly this tells me it’s still not cost effective. The issue is the system power consumption. My GPU only uses 120w, but to have the computer on and running the power at my outlet is 250-300. I like to use crypto compare because I can plug in the whole system power consumption.

That being said, these new cards might get ludicrously higher hash rates making them viable in a single GPU setup.

Update:

We are so excited to inform all of you that we are currently at ProductHunt, we appreciated if you can help us getting top of the list by upvoting us if you used and think would be helpful for others too:

https://www.producthunt.com/posts/deepnamer

When I have to make figures like these I use Inkscape, an open-source vector graphics software. If you use the grids and "snap to grid" mode, you can quickly make those types of figures. Even though it's vector graphics you can also import images like those input/output ones and then use skew transformations to get this result.

Processing is used for digital artwork:

https://processing.org/

I think they have some neural network code to do object detection.

I don't think it will do what you want though.

I dont think you would need deep learning for this. Basically all you need is a state machine that responds to commands by navigating to certain webpages and maybe presents info to facilitate the process. I think a good example to look at for inspiration to accomplish what you are asking might be Mycroft https://mycroft.ai/ or telegrams api https://github.com/python-telegram-bot/python-telegram-bot using deep neural nets for this would be way overkill.

There's an open source platform called Mycroft which is the open source version of Alexa/Siri. You may want to give that a look. They have a partnership with a company called Mimic that does text to speech. https://mycroft.ai/documentation/mimic/#why-should-i-use-mimic-instead-of-other-open-source-tts-tools

Not sure if they meet your requirements - my guess is you would need more training data to get a good TTS output but this might be a good starting point.

From their website:

> A language empowering everyone to build reliable and efficient software. > ... > Rust’s rich type system and ownership model guarantee memory-safety and thread-safety — enable you to eliminate many classes of bugs at compile-time.

I don't know much neither, but it could prevent overflow issues. (could, not will, because apparently some people managed to fail that by explicitly using unsafe code.)

I agree. I was just introduced to this person does not exist and then on the heels of it, from the great bowels of AI hell, this cat does not exist.

 

You know at some point we're gonna have to draw the line...

Programming is a foundation such as statistics and linear algebra for ML. These have to be learned sometime. At least for a solid foundation. But you can also start with high-level ML and DL courses where the concepts are explained and see if this is something for you at all and then work out the basics bit by bit to understand ML in depth. At some point you will come to the point where you need to be more involved with programming, at least if you also want to work practically with ML and not just with theory. With me it was at least like that.

​

Translated with www.DeepL.com/Translator

I deleted my comment as mods may not like it or it may breach certain ToS. Here it is via PM:

https://www.seedr.cc/zip/61838537?st=978d5693bf27b1e3c1819533d00eb35a92b414c77ba9578a9baee98acb4c9e4d&e=1562056288

The original Youtube link I found it from is here: https://www.youtube.com/redirect?event=comments&stzid=UgydhF3s6Q1cIqFeRtB4AaABAg&redir_token=ah_O8TZKpL9a2wcFAAPystGGSNp8MTU2MjgwNjIwM0AxNTYyNzE5ODAz&q=https%3A%2F%2Fwww.seedr.cc%2Fzip%2F61838537%3Fst%3D978d5693bf27b1e3c1819533d00eb35a...

It's the very last comment that had that link.

Good luck.

First, you need to format your code correctly. Not only is everyone online incredibly pedantic about this since code becomes very difficult to read without proper formatting, but Python's syntax is dependent on whitespace, so the code you've pasted will (a) not work as-is and (b) will behave differently depending on where you put the whitespaces.

As for your specific question, you're constructing a neural network using PyTorch. You're creating various layers, and connecting them sequentially to the next one by taking the output of one layer and passing them to the next (i.e., what you're doing with that `x` variable). At a high-level, you're just constructing a pretty simple convolutional neural network. Of course, the code you've presented here won't work since `x` was called before it was defined (among a few other issues), but I'm guessing that's due to some code you've omitted.

Walking through this, line by line, is, frankly, not a good way of understanding what you're doing here, since you're arguably doing the same thing, line by line. You're better off understanding what you're trying to accomplish as a whole, in which case the PyTorch docs will be your best resource. You can probably get through the introduction pretty quickly, then you can look through the docs to figure out what each of these functions are doing specifically.

Sure of course that's possible. It's something that's common in NLP: You take a series of words and make a classification on it.

The class of problem you're tackling is likely sequence classification. So you will need to define what a "sequence" in your data would look like. Once you pack your data into sequences you will probably get a tensor with shape (batch size, sequence length, number of features). This is appropriate for most LSTM implementations. The paper you are studying might have mentioned how this sequence is created, and what its length is.

The LSTM on its own will likely return to you three tensors, one will be output, two will be the hidden states. Once you have fed your sequence through in-full, you will feed the output through a feed forward network.

You can read more about the LSTM specifically for Keras here. Pytorch, while more complicated, also explains how certain networks work pretty well. Here is their LSTM page.

Again, good luck!

You cam but it's an irrelevant design decision. I'd create a separate component because the dataloader should handle data loading, not processing into patches.

https://pytorch.org/docs/stable/generated/torch.narrow.html#torch.narrow

Most deep learning frameworks provide pretrained models, have a look at torchvision. I am not sure what use weights have in text format, but it should be relatively easy to convert them, e. g.:

import torchvision.models as models alexnet = models.alexnet() for name, parameter in alexnet.named_parameters(): with open(f"{name}.txt", "w") as file: file.write(str(parameter.tolist()))

How does this differ from torchvision and the torchvision model zoo?

Not knocking your project, BTW, it seems neat. Just curious, seeing as torchvision is already pretty well-established.

And if you are using Pytoch, there are also lots of good tutorials. I suggest you take a look at this one: this Pytorch tutorial

Frameworks like PyTorch and Tensorflow have an Autograd (automatic differentiation) engine, meaning that any custom loss function that you implement using PyTorch/Tensorflow operations will have a backward implementation automatically (because the gradient of any of those operations already has an implementation). You only have to implement the forward pass. See https://discuss.pytorch.org/t/custom-loss-functions/29387

For more about PyTorch' automatic differentiation engine, see https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html

In pytorch this is quite easy You should follow this tutorial on how to use custom backbones with faster-rcnn and then tune end-to-end.

ONNX isn't going to work for every model. I'd encourage you to check out exporting PyTorch models to TorchScript for production inference. More info: https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html

That is rather ironic indeed. Is the reason for that the fact that my dataset is so skewed and I am giving the same weight for both classes?

I am working with pytorch and I guess this is the equivalent for the sigmoid cross-entropy that you mentioned (only exists in TF)
https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss

​

The idea is to change my BCE with this loss function alongside with Dice, right?

Yes, they absolutely can. Here's a tutorial on LSTMs in pytorch.

And here are the docs. Notice that if you use the batch_first flag, your inputs are of shape batch, sequence_length, num_features and your hidden units will be of shape batch, sequence_length, num_hidden.

Then, you can use a Linear layer (just a matrix multiplication) that inputs your hidden units and outputs 1 neuron. With a sigmoid and BCELoss, this will be the probability of your binary variable.

Official tutorials is a good start. I also followed some courses (e.g. CS231n) homework, and tried to implement algorithms on small datasets to force myself to practice. It helps a lot to see what others implementing algos, and then try to write your own from scratch.

In pytorch you need to wrap your model using <code>torch.distributed.dataparallel</code> to take advantage of multiple devices.

I have never heard of TCC, but looking at the documentation it seems it is a headless mode to make the card run compute only without the graphics bit. Tesla refers to NVIDIA datacenters GPU mainly the V100 which does not have video outputs. It also seems to be Windows specific. I only use linux in my day to day. In the past, I did some experiment on Windows and if you use a GPU for compute, Windows can think the driver is not responding an crash, this seems to address this problem by disabling the video output of the GPU forcing windows to use something else for the GUI.

I think you need embedding text. If you're new to use text data as input, I recommend these links.

https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html

Good luck :)

Here are resources. I haven't had the chance to see google's docs, but these are the best I've seen to date.

https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html

You can enforce a floating point type, or at least try to enforce a certain precision. Also since most training/inference is now done on CUDA, it's easy to enforce your precision. Even the speed of training is heavily dependent on what precision you're using, so there's a lot of interest in enforcing these types.

You can see that these tensors here have particular precisions: https://pytorch.org/docs/stable/tensors.html

The POSIX standard has a specification for floats as well. I think you are thinking of cases like where you declare float x; in C and that will vary depending on your hardware/arch, but you can enforce say, a 32 bit float by checking that your compiler supports IEC 60559, which will guarantee a particular 32 bit float.

If you want, you can essentially guarantee float behavior, which is pretty important because high precision arithmetic is very error prone and being able to control as much of it as possible is critical for testing and optimization.

If you let your floating point types get defined by hardware, you'd end up with different weights. If you force a particular width even where the hardware doesn't support it, you'd likely just end up with much slower training.

I haven't touched the Pytorch source in a while and never really even looked at how they implemented tensors so I'm not sure exactly how things are done there, but you can definitely enforce some sort of consistency which will lead to identical weights even with rounding errors.

I use PyTorch: https://pytorch.org/tutorials/

For new folks joining my research group I recommend doing basically all of the "Getting started" tutorials, and then in the "text" section, doing everything except "Chatbot tutorial". Also useful in "Deep Learning for NLP with pytorch" will be "Sequence models and Long Short-term memory networks".

Even if some resources don't seem directly related, implementing many RNN's will make things a lot clearer to you. This is sort of a more research-oriented approach, but you likely won't find one super-easy copy and paste solution to your problem.

A hardware description, as well as the way users will be expected to use it, would be helpful.

• Nvidia GPUs? - Nvidia's NGC catalog of docker images is extremely helpful here - they contain already configured compatible combinations of libraries (cuda/pytorch/etc) and can save weeks of fighting to configure them yourself.

• Multiple users working on it at the same time? Jupyter Hub might be useful.

I agree that keeping that LTS OS is a good idea. If/when people think they need a newer version of ubuntu, like some Ubuntu 21.10 dependency, it's possibly they could get by with Ubuntu 21.10 docker images. Or a VM - but that gets painful to share GPU cards with VMs; and painful on your RAM if many people run a full VM at the same time.

I recently started reading this*, has been pretty good so far. I have very minimal Python skills but I have been able to follow along with it. It focuses on TF2 with Python and has a chapter dedicated to CNNs.

*'Deep Learning with TensorFlow 2 and Keras: Regression, ConvNets, GANs, RNNs, NLP, and more with TensorFlow 2 and the Keras API' by Antonio Gulli, Amita Kapoor, Sujit Pal

Do not listen to Life_Breadth. Ridiculous advice. Unless you are actually interested in pursuing a PhD. Do not make a decision like that simply because some stranger on the internet told you it was required to do work in a field that has many examples of self taught individuals. Personally, I think your best bet is to read the new Fast.ai book. In the later chapters, it breaks down complex ideas from papers and explains them in code. Here it is, Deep Learning for Coders with Fastai and PyTorch.

I'm not seeing any other differences except windows pro. Its the zephyrus M vs zephyrus S on amazon:

https://www.amazon.com/Zephyrus-Portable-GeForce-i7-9750H-GU502GW-AH76/dp/B07VF88TMR/ref=sr_1_4?keywords=ASUS%2BZephyrus%2BS%2BGX502&amp;qid=1577405055&amp;s=pc&amp;sr=1-4&amp;th=1

https://www.amazon.com/ASUS-Zephyrus-Portable-i7-9750H-GX502GW-XB76/dp/B07SN353GT/ref=sr_1_3?keywords=ASUS+Zephyrus+S+GX502&amp;qid=1577405055&amp;s=pc&amp;sr=1-3