Their business model is a paid professional version and after some years, they put the latest features in the Open Source version which is called "((OTRS)) Community Edition". If got compare the version numbers, you'll notice that the paid version is OTRS 8 while the Community Edition is OTRS 6. You can find it here https://community.otrs.com/. For your use case, as mentioned, have a look at the customer authentication backend database (will require a bit of coding with Magento API) , dynamic fields, process tickets and webhooks. I'm sure you can nicely integrate OTRS with Magento and you'll get a powerful toolchain.
We've installed OTRS Community Edition. Free, an important feature. OTRS can do lots of other stuff as well, like workflows, but those pieces seem overly complicated. OTRS works well for us.