Hi HN - Justin, Scott, and Barak here. We're excited to introduce Helicone (
https://www.helicone.ai) an open-source logging solution for OpenAi applications. Helicone's one-line integration logs the prompts, completions, latencies, and costs of your OpenAI requests. It currently works with GPT, and can be integrated with one line of code. There’s a demo at
https://www.helicone.ai/video.
Helicone's core technology is a proxy that routes all your OpenAI requests through our edge-deployed Cloudflare Workers. These workers are incredibly reliable and cause no discernible latency impact in production environments. As a proxy, we offer more than just observability: we provide caching and prompt formatting, and we'll soon add user rate limiting and model provider back off to make sure your app is still up when OpenAI is down.
Our web application then provides insights into key metrics, such as which users are disproportionately driving costs and what is the token usage broken down by prompts. You can filter this data based on custom logic and export it to other destinations.
Getting started with Helicone is quick and easy, regardless of the OpenAI SDK you use. Our proxy-based solution does not require a third party package—simply change your request's base URL from https://api.openai.com/v1 to https://oai.hconeai.com/v1. Helicone can be integrated with LangChain, LLama Index, and all other OpenAI native libraries. (https://docs.helicone.ai/quickstart/integrate-in-one-line-of...)
We have exciting new features coming up, one of which is an API to log user feedback. For instance, if you're developing a tool like GitHub Copilot, you can log when a user accepted or rejected a suggestion. Helicone will then aggregate your result quality into metrics and make finetuning suggestions for when you can save costs or improve performance.
Before launching Helicone, we developed several projects with GPT-3, including airapbattle.com, tabletalk.ai, and dreamsubmarine.com. For each project, we used a beta version of Helicone which gave us instant visibility into user engagement and result quality issues. As we talked to more builders and companies, we realized they were spending too much time building in-house solutions like this and that existing analytics products were not tailored to inference endpoints like GPT-3.
Helicone is developed under the Common Clause V1.0 w/ Apache 2.0 license so that you can use Helicone within your own infrastructure. If you do not want to self-host, we provide a hosted solution with 1k requests free per month to try our product. If you exceed that we offer a paid subscription as well, and you can view our pricing at https://www.helicone.ai/pricing.
We're thrilled to introduce Helicone to the HackerNews community and would love to hear your thoughts, ideas, and experiences related to LLM logging and analytics. We're eager to engage in meaningful discussions, so please don't hesitate to share your insights and feedback with us!
One thing I would really like to be storing with my requests is the template and parameters that created the concrete prompt. (This gets a little confusing with ChatGPT APIs, since the prompt is a sequence of messages.) Custom Properties allow a little metadata, but not a big blob like a template. I see there's a way to have Helicone do the template substitution, but I don't want that, I have very particular templating desires. But I _do_ want to be able to understand how the prompt was constructed. There's probably some danger on the client side that I would send data that did not inform the prompt construction, and balloon storage or cause privacy issues, so there is some danger to this feature.
Backoffs and other rate limiting sounds great. It would be great to put in a maximum cost for a user or app and then have the proxy block the user once that was reached as a kind of firewall of overuse.
Note your homepage doesn't have a <title> tag.