Costly Delusions and Self-Improving Engines
The best LLM solutions don't start out smart, they start out mediocre and learn fast. Believing you can "one shot" your problems will lead to frustration, a month of your engineer's time, and $25K in runaway spending. With nothing to show.
We've failed many times. Not every problem can or should be solved with an LLM. Too expensive. Wrong problem. Improper tool. By abusing this new technology in every which way, without even knowing the Art of the Possible, we've run into plenty of roadblocks.
So we started a framework to use across all our companies, and teach today:
Create scoped problems with defined positive outcomes
Allow for initial 50th percentile success rates
Generate enough data to make compelling improvements
Implement a tight feedback loop
The solution will skyrocket from the 50th percentile to the 99th percentile. Cheaply, too. If done correctly. The issue is, this takes time. You can't "one shot" a proper solution to your problem. But you can make dogged, incremental, constant progress.
Take email marketing. Let's say you want to start a campaign for your users. Does this fit our framework?
1. Scoped problem with defined positive outcomes
Let's create a marketing campaign for onboarding users that will raise free trial conversion from 1.25% to 2.5%.
If we don't make our goal clear (increase the conversion rate), or perhaps become vague (are we facilitating this over sms messaging? video marketing? blogs?), we will fail at the next few steps. Have this clear target in sight, something the LLM now knows to hurtle toward. Without specificity, the entire system will come crashing down.
2. You can afford an initial 50th percentile success rate
OK so 50th percentile is pretty bad. Of course we want to do better, but let's assume we can start on the low side. In this context, our email marketing campaign will have middling results. 5% open rate compared to your average of 25%. In other scenarios, we would kill the campaign rather quickly.
But remember, the LLM won't have the human context you and I possess. You should plug in as much data as you can when kicking off this project. Previous campaigns, personas, anything. But, an LLM can never have all the worldly context you have about your business. From knowing your taste to what campaigns have subtly influenced you to the reason you started this company and its core mission...you can hardly convey all of this to another person efficiently. Now imagine doing that to a bunch of sand particles mixed with silicon.
So be patient! It won't work out of the gate. In fact, expect the LLM to flounder. This is expected. We are setting a baseline. Use this as a jumping off point to start collecting data.
3. There is enough data to generate compelling improvements
This goes back to choosing the right problem.
Using an LLM for a marketing campaign like this will be useful if we have enough users signing up. This would be the wrong problem if you are adding 1-2 users a month (why are you focusing on conversions?? Increase your top of funnel!). But if you are adding 1-2 users an hour, you have enough signal to work with.
If you can't generate compelling data directly related to your goal, LLMs are not the right fit for your problem.
4. You can implement a feedback loop
How can we take our problem, our data, and our LLM and create a self-improving system?
Figure out a way to do two things:
Turn the LLM's outputs and data into inputs
Provide directional guidance
You want to take the marketing campaign the LLM has started building, and make it recursive. Take the collateral and the data generated, then throw it back into step 3, the data machine. Do this autonomously and as often as you can. Keep the feedback loop and data you provide scoped.
For example, let's say the LLM comes up with a new onboarding email, which links to a Getting Started guide.
Figure out how you can tape your systems together so you can take the data from the email and put it into the LLM, asking it to interpret the data and improve the system. For example, take the following data, pipe it into your database, and trigger a daily improvement prompt:
email content
click rates
responses
user behavior and attributes
LLMs will go the right direction for the email campaign maybe 80% of the time. The other 20%? You'll need to step in and make some tweaks. Say it will start interpreting responses as a positive signal...
If a user replies "TAKE ME OFF YOUR LIST," a naive LLM just sees engagement. It thinks, "A response is a response! Let's send more emails like that one." You have to guide the model on what great looks like, and compound to there.
The perfect prompt doesn't exist, and chasing it will only burn through your cash and patience. Accept that your first deployment is going to be mediocre. That is the whole point. Get the 50th percentile version out the door. Stop looking for a magic trick, and focus on building a self-improving engine.
Here are some more tips:
Storing data is wildly cheap nowadays. Try to store everything.
Make the human feedback loop ergonomic. Have a dedicated Slack channel for you to send feedback.
Explicitly tell the LLM they are allowed to delete. They have a disposition (much like humans) to add, when in fact deletion may be the best option.