This: >"how can we process language automatically?"
Requires this: >"how does language work?"
Science has never worked in the opposite. The reason you are even sitting on a computer reading this is because centuries ago a lot of really smart people came and gave us the laws of electromagnetism, calculus and understanding of chemistry to build electronic circuits and the silicon transistor which lead to modern day computer engineering. There is no way we could have engineered modern technology without major scientific and mathematical theories from the prior centuries.
> Those benchmarks are not about linguistics, they're about NLP.
If this is their mindset, then I'm sorry this is pure pseudoscience. It is a dead end. They are free to continue dumping billions in this direction.
Major silicon valley companies have poured a lot of money to compete on these benchmarks. There is a rat race among them to get human level language for their Google Duplex, Apple Siri, Amazon Alexa, Google Assistant, Microsoft Cortana, Facebook AI, Samsung's Bixby, Alibaba's AI.
IBM dumped a lot of money on Watson to beat humans at Jeopardy!, nothing came out of it. Watson was a total shit show. IBM dumped a lot of money into DeepBlue, their brute-forced chess playing AI. 20 years later, Google did the same with AlphaGO. We didn't get anything useful from these marketing ploys. We aren't doing anything fundamentally different.
Silicon Valley didn't anything learn from IBM's mistakes.
>recent developments in NLP have push the field of linguistics forward in ways never though before. So we (in my team at least) greatly appreciate the work coming out of the NLP community.
Probably because it's not a mature science yet. You haven't had enough exceptional minds like Einstiens, Newtons, Turings, Darwins come in and build a complete foundation yet. Whether or not NLP is useful, only the test of time will tell.
I'm not sure how credible Noam Chomsky is with linguistics, but I whole heartedly agree with his sentiments. Peter Norvig of Google seems to be in this pseudoscience bucket. This argument is a bit outdated, the modern form can be obtained by replacing "statistics" with "deep neural networks".
In the end, this is a bad way of conducting research, I've never seen this work in contemporary industrial research. If Boeing did things like this, we'd have an incredibly high death toll. The self driving car industry seems to be committed to doing things this way, they are failing. The prominent folks from big institutes seem to be lying to the public. Your research group may find it useful to conduct science this way, but I highly doubt anything substantial will come out of it in the next 10-20 years. I've never seen it work, it's too loosey goosey.
You should probably read a book on the history of science/physics. I recommend this one, even though it's a bit mathy.