Why code-testing startup Nova AI uses open source LLMs more than OpenAI

It’s widely recognized in the software development community that developers should not be solely responsible for testing the code they create. This principle stems from practical considerations — most developers tend to dislike testing tasks, and effective auditing protocols dictate that the individuals performing the work should not also be responsible for its verification.

Consequently, there has been a growing emphasis on code testing across various dimensions, including usability, language- or task-specific tests, and end-to-end testing, within the realm of generative AI startups. TechCrunch frequently features startups like Antithesis (which raised $47 million), CodiumAI (which raised $11 million), and QA Wolf (which raised $20 million), each focusing on different aspects of code testing. New entrants, such as the recently graduated Y Combinator startup Momentic, continue to emerge regularly.

Among these startups is Nova AI, a year-old company that graduated from the Unusual Academy accelerator and secured a $1 million pre-seed funding round. Nova AI distinguishes itself by targeting mid-size to large enterprises with complex codebases and urgent testing needs, defying conventional Silicon Valley startup norms. While the typical Y Combinator approach advocates starting small, Nova AI is catering to enterprises with immediate testing requirements. Although Smith, a representative of Nova AI, declined to disclose specific customers using or testing their product, they mentioned that their clientele primarily consists of late-stage (Series C or beyond) venture-backed startups in sectors like e-commerce, fintech, and consumer products, with a strong emphasis on demanding user experiences where downtime can incur significant costs.

Nova AI’s technology analyzes its clients’ code to automatically generate tests using GenAI. This approach is specifically tailored for environments characterized by continuous integration and continuous delivery/deployment (CI/CD), where engineers frequently deploy updates to their production code.

The inspiration for Nova AI stemmed from the firsthand experiences of its founders, Smith and Jeffrey Shih, who previously worked as engineers at prominent tech companies. Smith, a former Googler, specialized in cloud-related teams focused on automation technology. Shih, who has worked at Meta (previously Facebook), Unity, and Microsoft, possesses expertise in AI, particularly synthetic data. The team has since expanded to include a third co-founder, AI data scientist Henry Li.

In a departure from the norm, Nova AI has chosen not to heavily rely on OpenAI’s GPT models. While many AI startups utilize OpenAI’s GPT, Nova AI minimizes its usage of OpenAI’s Chat GPT-4 and ensures that no customer data is shared with OpenAI.

Despite OpenAI’s assurances that the data of those on a paid business plan is not utilized for training its models, many enterprises remain skeptical. Smith reveals that large enterprises, in particular, express reluctance to have their data incorporated into OpenAI’s systems. “When we’re in discussions with large enterprises, they express concerns about their data being shared with OpenAI,” Smith explains.

This wariness toward OpenAI is not unique to engineering teams within large companies. OpenAI is currently facing multiple lawsuits from individuals and organizations who oppose the use of their work for model training or believe their work has been included, without authorization or compensation, in OpenAI’s outputs.

Nova AI has adopted a different approach, leaning heavily on open-source models such as Llama, developed by Meta, and StarCoder from the BigCoder community (developed by ServiceNow and Hugging Face). Additionally, Nova AI is actively developing its own models. While they have tested Google’s Gemma and observed positive outcomes, they have yet to incorporate it into their customer offerings.

For example, Smith explains that while OpenAI provides models for vector embeddings, Nova AI opts to utilize open-source alternatives for this purpose, specifically on the customer’s source code. They utilize OpenAI tools primarily for code generation and labeling tasks, taking precautions to avoid sending any customer data to OpenAI.

“In this scenario, instead of relying on OpenAI’s embedding models, we utilize our own open-source embedding models. This approach ensures that we’re not simply sending every file to OpenAI,” Smith elaborated.

While the decision to avoid sending customer data to OpenAI alleviates concerns among cautious enterprises, Smith has discovered that open-source AI models are not only cost-effective but also more than capable of fulfilling specific tasks. In the case of Nova AI, these models excel at generating tests.

“The open LLM industry has demonstrated its ability to outperform GPT-4 and other major domain providers, particularly in focused applications,” Smith remarked. “We don’t need to deploy a massive model capable of predicting what your grandmother wants for her birthday. Our focus is on writing tests, and our models are finely tuned for that purpose.”

Furthermore, open-source AI models are evolving rapidly. For example, Meta recently unveiled a new iteration of Llama that has garnered praise within technology circles. This development may prompt more AI startups to explore alternatives to OpenAI.