ML EXPLAINED

Tag: technology

  • Make Small PRs And Read Your Code

    It has been quite some time since I’ve written here. Things have been busy in life and generally I haven’t had any motivation to share knowledge or concepts, I see that the content I put out is not bad, but there is better out there and there is less and less to contribute also in the age of AI.

    But from my extensive experience of working on AI projects, making an AI sales/account manager and using coding tools like Cursor, Claude Code or Codex extensively professionally or in personal projects like Detective Kai, one thing I’ve understood is to keep the scope of your changes small.

    It’s very tempting to let the coding models implement every idea you’ve had and see them in action, and this temptation is what makes you lose grasp of the project. Once a change is around 2k or more lines of code, then you’re not in the driver’s seat but more of a passenger in the ride. Rather than understanding the full picture, you start relying on the tool to explain the nitty-gritty of the system. And when this starts happening, it’s not good. This is what leads to debugging hell, where you know something is wrong but cannot fix it as you now don’t understand the systems you’ve built.

    The modern AI evangelist will argue “skill issue” and just prompt the system bro, but I’ve used Fable and GPT-5.6-sol at max thinking as well, but the problem remains the same, these models don’t solve for the longevity of the project, but they solve immediate problems which you present. It could be due to the post-training RL being done, but they want to solve the immediate issue and not care about how it affects the system in the long run. Only humans care about this. You care about whether the elements are modular or not, whether the complexity of the solution is in line with the gains it brings, etc. These things are important, and while the intelligence of these models has grown, their context hasn’t, and we know that increasing the context is not trivial, so on larger projects, which anyone who goes beyond a POC will eventually face, it’s not possible to rely on these models alone to carry you, and earlier if the scope of your work was larger, it was okay. Yes, no one could have realistically reviewed your 5k lines of code change PR, but you understood it. Now with AI pumping out these humongous PRs, the trouble is even you don’t understand it. So my suggestion is to scope it down and read your own code.

  • How does ChatGPT remember? LLM Memory Explained.

    In the fascinating world of conversational AI, the ability of systems like ChatGPT to remember and refer back to earlier parts of a conversation is nothing short of magic. But how does this seemingly simple act of recollection work under the hood? Let’s dive into the concept of memory in large language models (LLMs) and uncover the mechanisms that enable these digital conversationalists to keep track of our chats.

    The Essence of Memory in Conversational AI

    Memory in conversational AI systems is about the ability to store and recall information from earlier interactions. This capability is crucial for maintaining the context and coherence of a conversation, allowing the LLM to reference past exchanges and build upon them meaningfully. This also gives the appearance that the LLM has intelligence when in reality they are stateless and have no inbuilt memory.

    LangChain, a framework for building conversational AI applications, highlights the importance of memory in these systems. It distinguishes between two fundamental actions that a memory system needs to support: reading and writing.

    What happens is that the LLM is passed an additional context of memory in addition to your input as a prompt so that it can process the information as if it had all the context from the get-go.

    Building Memory into Conversational Systems

    The development of an effective memory system involves two key design decisions: how the state is stored and how it is queried.

    Storing: The Backbone of Memory

    Underneath any memory system lies a history of all chat interactions. These can range from simple in-memory lists to sophisticated persistent databases. Storage is simple, you can store all past conversations in a database. You can either store them as simple text documents or use a vector database and store them as embeddings.

    Querying: The Brain of Memory

    Storing chat messages is only one part of the equation. The real magic happens in the querying phase, where data structures and algorithms work together to present a view of the message history that is most useful for the current context. This might involve returning the most recent messages, summarizing past interactions, or extracting and focusing on specific entities mentioned in the conversation.

    Practical Implementation with LangChain

    Here we will take a look at one way to store memory using LangChain. 

    from langchain.memory import ConversationBufferMemory

    memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

    Now you can attach this memory to any LLM chain and it will add the entire previous conversations as context to the LLM after each chain invoke. The advantage of using this kind of memory is that its simple to implement. The disadvantage is that in longer conversations you’re passing more tokens and the input prompt size explodes, meaning slower response and if you’re using paid models like GPT-4, then costs also increase.

    Conclusion

    The ability of systems like ChatGPT to remember past interactions is a cornerstone of effective chatbots. By leveraging sophisticated memory systems, developers can create applications that not only understand the current context but can also draw on previous exchanges to provide more coherent and engaging responses. As we continue to push the boundaries of what conversational AI can achieve, the exploration and enhancement of memory mechanisms will remain a critical area of focus.