Are you trying to burn them in? Or add them as tracks?
Handbrake can add subtitles either as tracks or burned in, but it will reencode the video.
Subler can add them to an existing mp4 as tracks.
There is a very similar program to MeGUI on the Mac called Hybrid, but I have only recently become aware of it. The dev also stopped updating it on the Mac late last year.
I have found that using a deband filter along with a deblock filter helps to reduce the blocky artifacts drastically. I used a image from the 0.2 version thread posted by u/Judas0001 because those artifacts were pretty noticeable there, and after applying the filters they were pretty much gone.
Adding a built in deband and deblock filters is not really necessary because you can load the merged frames in something like Hybrid for any post processing you may need, but I wanted to mention it because I found that they give very good results, and in the future may be nice as an option.
For what purpose do you want to encode in h.265? for archiving? The GPU/hardware encoding is only interesting for live encoding, like streaming, but even then it has its disadvantages. The quality of hardware encoding can not match the quality of CPU encoding. So for archiving your only option is a faster CPU.
If you want to archive your videos or save some space then use the cpu encoder x265. There are many tools for this around like Hybrid. Your CPU is not the fastest and i think it should be time for an upgrade now. - Hey, btw. did you hear that AMD has just released the 2nd gen Zen cpus now? :D
If you are OK with a higher bitrate but still need a fast encoding then use the fast or faster preset on the x265 encoder.
Why do you even convert?
Unless the contained video or audio format isn't supported by your playback device, you should just remux it (means change the container/wrapper format from MKV to MP4), unless of course you actually need to reduce the filesize.
You can remux with Xmedia Recode (https://xmedia-recode.de/en/download.html) or ffMPEG or Hybrid (http://www.selur.de/downloads). In Xmedia you simply set Mode = Copy on the Video and the Audio tab. This only takes a few seconds, and the video quality of the MP4 file will be exactly the same as the Matroska source file... because nothing was converted.
In this case, the video is OK, so you don't want to re-encode the video, only the audio. I therefore recommend that you use the program Xmedia Recode (http://www.xmedia-recode.de/en/download.html). This tool is Windows only, so if you're a Mac or Linux user, you can use Hybrid (http://www.selur.de - The installer file is huge, but the program is just as reliable as Xmedia).
Start Xmedia Recode. Load the MKV. Set the output format to Custom & MP4 and define a destination path.
Now click on the Video tab (highlight the file if it doesn't appear). Set the Mode field to "Copy".
On the Audio you have the same Copy option, but this is obviously where your problem lies, so choose Mode = "Convert" instead & set Codec to AAC or E-AAC or another simpler, widely supported audio format like MP3
If the source file has embedded subtitles, or if you want to add an external SRT file, you also click on the Subtitle tab and use the "play" icon to add the wanted subs to the output file.
Finally click on Add to queue > Encode - and a few seconds later you have a playable MP4 video file that should work. No video quality is lost in this process.
If you meant what you said about "do[ing] this with the least loss of quality", I suggest you first check if you even have to convert anything. If the AVI or MKV-contained video format is H264 (also called AVC, mp4v, x264 or mpeg4), then you will simply degrade your video quality for no reason, if you pull it through Handbrake. Same goes for H.265 aka. HEVC.
Try the options that allow video passthrough (aka. remuxing / direct streamcopy etc.) such as Permute or iffMPEG. Then you both save time and quality. For PC there is Xmedia Recode, and for all desktop OS's you can use Hybrid.
These are some basic elements of remastering/restoring old footage
The MOST important thing is trying to find the highest quality footage you can (searching on YouTube or surfing the web to find good footage is your best bet, but make sure it's the HIGHEST QUALITY footage you can find). There's no point in 'remastering' ultra-crappy footage; that's like dumping glitter onto a pile of dirt. There's an old adage that goes hand-in-hand with remastering footage - "Garbage in, garbage out". Basically, if the original footage of looks atrocious, there's nothing substantial you can do about it; so don't waste your time with it.
Color Correction (contrast, brightness, saturation, etc.) is very simple and can be done in most video editing software. Despite its simplicity, it is incredibly important because the colors in old footage are typically washed out and ugly. I recommend watching color correction tutorials that correlate with the editing software you're using. (e.g. I use Sony Vegas, so I can just search 'sony vegas color correction tutorial')
Post-processing (de-noising, de-blocking, grain reduction, sharpening, etc.) is yet another extremely important thing. Just like with color correction, post-processing tools are present in most video editing software.
Motion Interpolation is a technical term for artificially increasing the frame-rate of a video. In this Dreams Never End remaster, the original non-remastered footage was at 25fps, but I used Hybrid to interpolate the video to 60fps. Hybrid is a rather complicated program, but there are plenty of tutorials that will show you how to use it to interpolate videos. Just search up 'Hybrid 60fps' (or something similar) on Youtube, and you'll have no trouble finding a tutorial.
If you would like a ready to go alternative you can give hybrid a go. Supports AVISynth + Vapoursynth with a ton of built in plugins.
You may have already done this but it may be worth mentioning as it seems to be an NTSC source, and for anyone who stumbles across this:
It may be telecined instead of interlaced, in which case you can recover the original 24fps video. In the deinterlaced video if you step frame by frame and see repeated frames (2 then 3), that is the case and you can use a IVTC filter instead.
You might want to look into another NVENC implementation in other software, such as Selur's excellent Hybrid software and Staxrip. They allow one to tune it out for an excellent compromise on throughput and quality.
With Hybrid in particular, I can even transcode to and from HEVC HDR content with NVENC.
Handbrake's configurability in this regard is very limited.
This this: http://www.selur.de/
It supports many Avisynth and Vapoursynth filters, including a few that do motion interpolation, if you have used SVP before it is based on an Avisynth plugin called InterFrame, the Vapoursynth filter "Interframe/SVP" in Hybrid is the same thing.
This are the settings I normally use.
Alternatively, SVP also has a program to convert a video to 60 fps called SVPcode, which is far easier to use, but Hybrid is far more complex and is what I would recommend having tried both.
If you have videos you want to store, like recorded gameplay or DVD/Blu-ray rips for example, compress everything with Handbrake, Hybrid or a FFMPEG command using X265 slow/slower and a CRF of around 19-23 depending on how much compression you want.
X265 slower CRF 23 should result in a good quality at a very low file size, but it's very slow.
Personally I would recommend using Handbrake for it's simplicity, but Hybrid have some really interesting features like AviSynth and VapourSynth filters, or the possibility to export to many different formats like webm for example, which Handbrake don't support even if you select the VP8 or VP9 codec, but it's way too complex if you only need a basic usage.
There is also this, which lowers the time it takes to upscale a video considerably, it still takes forever though. The only problem is that it only works on Nvidia GPUs because it uses Waifu2x-caffe.
If the original video's quality is decent enough, I'd rather use NNEDI3CL along with a denoise filter (DFTTest usually), and maybe AWarpSharp2 in a subtle way. If the original video doesn't look terrible, the difference in quality won't be that big compared to Waifu2x, but the difference in time is massive. I use Hybrid to apply those Vapoursynth filters in a easy way, since I'm not a big fan of using console commands and would rather have a GUI.
just drop your video and in video use x264 then in x264 tab use constant rate factor 1-pass as encoding mode, quantization factor = 17.5, avcprofile high, 5.0, preset medium (apply by hitting the arrow on the right) and leave everything else as it is. as for the audio, you can also reencode it or leave it as it is.
if you want to check how the transcoding is going check the %temp% folder and open it with a video player.
FYI I believe you have Boot Camp pre-installed on your Mac (https://support.apple.com/en-us/HT201468). That should allow you to boot Windows and install any Windows application.
But AVIdemux or Hybrid (http://www.selur.de) also has a video remuxing option (sometimes referred to as "video passthrough"), if you prefer to use MacOS software only.
Different encoding program.
http://www.selur.de/
I've had much better results with hybrid than any other when using HVENC. It still doesn't compare to x265 though. Selur will also help with bugs and such.
Does it actually stop working, or would it perhaps work, if you wait a minute or two?
If it simply doesn't work, there is also a tool called Hybrid, which is an equally good alternative GUI for ffMPEG: http://www.selur.de/downloads ... Just bear in mind that the installer i 261 mb. After installation, Hybrid takes up nearly 600 mb on your hard drive.
To use it, click the icon with an arrow pointing to the right near the top of the window to load the file. Then set the parameter Video handling to "passthrough". You can try the same with the audio handling, but perhaps it's best to try setting it to "Custom" first. Then click the Audio tab that appears and check "Audio encoding options" > aac. Finally, set a destination path + filename, and then click on the Start queue icon.
As a PC user I use XMedia Recode for this. A multi-platform alternative is Hybrid. Moving the contained video and audio streams in and out of the MKV without reeoncoding is extremely simple in both tools. XMedia has an extremely simple NLE function. There is also the official Matroska toolkit called MKVtollnix (cross platform), which probably offers even more features.
A note on conversion: If you receive an MKV containing FFV1 video, I would avoid converting to ProRes, as ProRes is a lossy format, while FFV1 is a future-proof lossless format recommended for long term preservation. It seems that Adobe Premiere can handle FFV1 via ffDirectShow, while other popular NLEs like FCP or SVP cannot. See: https://en.wikipedia.org/wiki/FFV1#Applications_supporting_FFV1