It's called a Risset or Shepard Tone. There is a risset tone generator that is used in the program Audiomulch. Basically it creates an acoustic illusion "constructed by creating a series of overlapping ascending or descending scales." The wiki article has several pop culture examples of these tones that might be what you're looking for.
I used one in a song once, thought i was having flashbacks when I clicked your link.
EDIT - there are some additional FX on the particular one in the clip you posted. The example used in the wiki article doesn't sound much like what's in your youtube clip, but the generator in Audiomulch can be made to sound exactly like it. I realize I've failed to tell you what you are remembering the sound from, but I got excited that I knew what it was so kind of forgot. It could be used in lots of different things really. Hope this was at least a little informative.
EDIT 2 - Basically rewrote my whole damn post because I'm acting like a n00b
Seems like you are looking for an arpeggiator with random note selection?
You can try one with Audiomulch, it has an arpeggiator module which allows you select the notes you would like to use and which octaves you would like them in, it works with virtual patch cables that you connect to an output.
I know there are probably others (Ableton, Reason, maybe even standalone programs etc...), I'm just familiar with that one.
You could also ask in /r/WeAreTheMusicMakers, who probably know of other programs.
I'm not sure if you mean the end result has visuals similar to the website, or the tool allows you to create the music in a style similar to this website. If you meant the former, I'm afraid I can't help.
If it's the latter, there's actually a lot of them. The artist in the link (Girl Talk) says he uses Adobe Audition to edit the individual tracks, then AudioMulch to assemble the music. AudioMulch is what you would be looking for.
You can also search for loop based editors, music trackers (Wiki page) or search for artists who seem to be doing this kind of thing and see if they've mentioned what they use in interviews.
Perhaps obviously, these things are geared toward musicians and actually selling the software, so they're pretty complicated if you don't quite know what you're trying to do. A very long time ago I remember playing with something called AcidPro that was easy to use, and they had a store or some such with lots of free samples to use.
He also uses Audiomulch which is some really techy shit. I think he went with ableton in this vid because he was only allotted 10 minutes to make the beat. This article sheds some more insight into his approach, but it's from 2003 -- when ableton was still pretty young -- so I don't think he was using it just yet. He also talks shit on Reason in that article which is pretty cool.
To be honest, I don't really have any knowledge beyond my own Ableton experience and these Girl Talk videos about Audiomulch. I think it's gonna be a pain to format all my samples for live shows no matter which program I use :/
Looks like it’s AudioMulch.
https://www.quora.com/What-software-does-Girl-Talk-use
Not custom software but looks pretty configurable so he probably customises it quite heavily. Definitely not your average noob Serato set up.
I agree with what Siapo said. I'd just add that Mulch is great for all of the reasons you mentioned because that's what it was designed for. Straight from the horse's mouth, it was never intended to be a production tool (see quote below from the developer).
Re external instruments etc my approach would be to record each instrument to a seperate file (it even has a 32 file recorder contraption, and you can have as many as you system memory will allow). Then I'd use Reaper to mix the tracks as you see fit.
​
BTW, I use Win 10 and virtual cables like kithchen_ace suggested still work .
​
From: http://www.rossbencina.com/projects :
"AudioMulch is software for musical performance, production and audio processing. It has a modular, non-linear workflow designed to support improvisation. Think of it as a musical instrument rather than a production or tracking tool. I created it for my own use back in 1998. Since then the user-base has grown. Users range from bedroom hobbyist musicians and experimental sound artists to high-profile performers."
Here's the documentation for the main looping device in Audiomulch:
http://www.audiomulch.com/help/contraption-ref/LiveLooper
You get optional syncing, three different modes (manual and two types of chaning) and 16 tracks - with a suitable midi controller this should be a pretty effective looping setup.
I've used Audacity (free) for simple production for many years. About 2 years ago, Girl Talk performed at my university and I met him and briefly saw his setup on stage during sound check. I noticed that he was using a program called Audiomulch (free demo). That led me to try the demo and I eventually bought the software (and the more recent upgrade) and am extremely happy with it. Some say it is too complicated but you can try the demo and decide for yourself. As a musician who has years of experience with a wide variety of audio software and hardware, I feel very comfortable with it and enjoy building and using my virtual audio contraptions. I still use other software to record and produce, usually REAPER (free).