You made huge claims using a non peer reviewed preprint with garbage statistics and abysmal experimental design where they put together 21 bikes and 4 race cars to bury openAI flagship models under the group trend and go to the press with it. I’m not going to go over all the flaws but all the performance drops happen when they spam the model with the same prompt several times and then suddenly add or remove information, while using greedy decoding which will cause artificial averaging artifacts. It’s context poisoning with extra steps i.e. not logic testing but prompt hacking.
This is Apple (that is falling behind in its AI research) attacking a competitor with fake FUD and doesn’t even count as research, which you’d know if you looked it up and saw you know, opinions of peers.
You’re just protecting an entrenched belief based on corporate slop so what would you do with peer reviewed anything? You didn’t bother to check the one you posted yourself.
Or you post corporate slop on purpose and now trying to turn the conversation away from that. Usually the case when someone conveniently bypasses absolutely all your arguments lol.
5 Conclusion
In this study, we investigate the capacity of LLMs,
with parameters varying from 7B to 200B, to com-
prehend logical rules. The observed performance
disparity between smaller and larger models indi-
cates that size alone does not guarantee a profound
understanding of logical constructs. While larger
models may show traces of semantic learning, their
outputs often lack logical validity when faced with
swapped logical predicates. Our findings suggest
that while LLMs may improve their logical reason-
ing performance through in-context learning and
methodologies such as COT, these enhancements
do not equate to a genuine understanding of logical
operations and definitions, nor do they necessarily
confer the capability for logical reasoning.
And still nothing peer reviewed to show?
Synethic benchmarks mean nothing. I don’t care how much context someone can store, when the context being stored is putting glue on pizza.
Again, I’m looking for some academic sources (doesn’t have to be stem, education would be preferred here) that the current tech is close to useful.
You made huge claims using a non peer reviewed preprint with garbage statistics and abysmal experimental design where they put together 21 bikes and 4 race cars to bury openAI flagship models under the group trend and go to the press with it. I’m not going to go over all the flaws but all the performance drops happen when they spam the model with the same prompt several times and then suddenly add or remove information, while using greedy decoding which will cause artificial averaging artifacts. It’s context poisoning with extra steps i.e. not logic testing but prompt hacking.
This is Apple (that is falling behind in its AI research) attacking a competitor with fake FUD and doesn’t even count as research, which you’d know if you looked it up and saw you know, opinions of peers.
You’re just protecting an entrenched belief based on corporate slop so what would you do with peer reviewed anything? You didn’t bother to check the one you posted yourself.
Or you post corporate slop on purpose and now trying to turn the conversation away from that. Usually the case when someone conveniently bypasses absolutely all your arguments lol.
Okay, here’s a non apple source since you want it.
https://arxiv.org/abs/2402.12091