GPT-4 Can Use Tools Now—That’s a Big Deal

Human children come out of the womb totally helpless except in one important way: they know how to use their parents as tools. 

Infant tool use is quite blunt at first: they cry loudly and incessantly whenever there’s a problem: “HUNGRY”, DIRTYY DIAPER!!!!”, “TIREEDDD!!!!”, and so on. They keep crying until their parent adequately diagnoses and resolves the issue through trial and error. 

As they get older, however, children ditch these crude initial methods and instead use language to skillfully manipulate their parents in ever more targeted and precise ways. Rather than simply becoming totally unconsolable because they see someone eating a cookie and want one for themselves they can now specify in precise language exactly what they want: “Can I have a cookie?”. Parents can then use their unique capabilities—the ability to walk, their height differential, their manual dexterity, and strength—to walk to the cookie jar, open it, select a cookie, and appropriately offer it up as tribute. This kind of tool use is a powerful method for intelligent beings with significant limitations to accomplish goals in the world.

In contrast to human children, large language models like GPT-4 were not created knowing how to use tools to accomplish their aims. This limited their capabilities significantly. Third-party libraries tried to implement this functionality—but the results were often slow and inconsistent.

Earlier this week, OpenAI built tool use right into the GPT API with an update called function calling. It’s a little like a child’s ability to ask their parents to help them with a task that they know they can’t do on their own. Except in this case, instead of parents, GPT can call out to external code, databases, or other APIs when it needs to.

Each function in function calling represents a tool that a GPT model can use when necessary, and GPT gets to decide which ones it wants to use and when. This instantly upgrades GPT capabilities—not because it can now do every task perfectly—but because it now knows how to ask for what it wants and get it.

Function calling works like this:

When you query a GPT model you can now send along with it a set of tools that the model can use if it needs to. For each tool you can specify a description of its capabilities (do math, call a SQL database, launch nuclear bombs) and instructions for how GPT can properly call each one if it wants to. Depending on the query, GPT can choose to respond directly, or instead request to use a tool. If GPT sends back a request to use a tool your code calls the tool and sends back the results to GPT for further processing, if necessary.

Tool use is important because, just like a four-year-old, GPT models currently have some glaring limitations: they’re horrible at math, they don’t have access to private data, they don't know about anything past 2021, they can’t use APIs, and more. In order to fix these problems OpenAI has harnessed GPT’s reasoning abilities to choose for itself when to use a tool to help it with a query that it knows might be difficult for it.

This matters for two big reasons: 

1. It makes GPT models significantly more powerful
2. It replaces some of the functionality of open-source libraries that do the same thing

GPT the tool user > GPT

Given all of the current hype about AI, you could be excused for forgetting that traditional software is actually very powerful. Right now, that power is mostly cut off from GPT models because it’s quite hard to integrate the two. If you could do that well, it would be a force multiplier on AI’s existing capabilities. That’s what makes function calling very exciting.