Home > Apps

Can ChatGPT pass the bar exam?

OpenAI's GPT-3.5 makes headlines by passing the bar exam.
Last Updated on September 8, 2023
Has ChatGPT passed the bar exam?
PC Guide is reader-supported. When you buy through links on our site, we may earn an affiliate commission. Read More
You can trust PC Guide: Our team of experts use a combination of independent consumer research, in-depth testing where appropriate - which will be flagged as such, and market analysis when recommending products, software and services. Find out how we test here.

Every day there seems to be a new story about ChatGPT. The latest headline surrounds the AI bot’s potential to pass a standardized law assessment. But, can ChatGPT pass the bar exam?

ChatGPT has been turning heads ever since its release in November last year. In the last 24 hours, Buzzfeed even announced it will be using the AI model to write some of its content – pretty big news!

The developers at OpenAI have been showing us what this language model can do, and it’s scary. With companies implementing these models to replace human writers and now its potential to pass exams, what more can we expect from AI in the future? Let’s explore whether or not this AI language model can pass the bar.

What is the bar exam?

The bar exam is the final assessment for someone looking to become a licensed lawyer. In order to practice law in a particular state in the US, one must pass the state’s specific bar exam.

The test has a reputation for being very difficult, with overall pass rates sometimes reaching as low as 30%.

If you’ve ever wondered what the exam looks like, you’re not alone. The bar exam questions can take two different forms, multiple choice, and essay. The tests are usually split across many separate sessions sometimes taking several hours to complete.

Did ChatGPT pass the bar exam?

Professors from Illinois Tech and Michigan State College of Law put GPT-3.5 up against the multistate bar exam (MBE). In their research paper, they found that the model “achieves a passing rate on two categories of the Bar” and achieved similar marks to human “test-takers” on another.

The results are outstanding, and they believe that GPT-3.5’s answers show that the model has a “general understanding of the legal domain”. If you are interested in reading the entire research paper, you can find it here.

So we know that GPT-3.5 can pass the bar, but what about other law exams? Well, other law professors at the University of Minnesota also tested out OpenAI’s language model on another exam scenario. They made Chat GPT take four assessments covering different courses in the university’s law degree. The results were fascinating.

The chatbot only managed to produce mediocre scores of just a C+ but still passes nonetheless. The findings also seem to suggest that ChatGPT would be advanced enough to potentially secure a degree in law if it were a student at the college.

Did GPT-4 pass the bar exam?

OpenAI’s latest model, GPT-4, not only got admitted into law school but also completely aced the Bar exam. For context, the multimodal model scored in the 88th percentile for the LSAT or Law School Admission Test. And, for the Bar? Well, this model scored in the top 90th percentile. This is a massive achievement considering OpenAI’s later model, GPT-3 could only score within the bottom 10% of test-takers.

Final thoughts

There’s no doubt about it, OpenAI’s language models are pretty clever tools. Its ability to pass exams is very clear and its recent passing mark on the bar has caught many by surprise. If you found this article interesting, be sure to read another article here.

Funmi joined PC Guide in November 2022, and was a driving force for the site's ChatGPT coverage. She has a wide knowledge of AI apps, gaming and consumer technology.