ML EXPLAINED

Tag: Mistral

  • Is Llama 3 Really Better Than Mistral?

    Is Llama 3 Really Better Than Mistral?

    With the recent launch of the much anticipated Llama 3, I decided to use both Mistral, which is one of the best small (7B) language models out there, and Llama 3, which according to its benchmark scores claims to outperform Mistral. But is it really better when it comes to using it as the LLM in your RAG applications? To test this, I put the same questions to both Mistral and Llama 3 and the results will surprise you.

    Link to Collab

    I created a RAG application using Ollama. In case you want to know how you can do it yourself, you can check out this post. I used the Elden Ring Wikipedia article as the document for contextual retrieval. I was using conversation buffer memory, which just passes the entire conversational history as context back to the language model. Furthermore, I asked the same question to both LLMs, and at the end we also asked the same question to the current king of LLMs, GPT-4. The question was –

    "How many awards did Elden Ring Win, and did it win Game of the year award ?"

    The entire prompt with the context was – 

    Be precise in your response. Given the context - Elden Ring winning Game of the Year
at the 23rd Game Developers Choice
AwardsSome reviewers criticized a number of the game's menu and accessibility systems.[84][85] Reviewers
complained about the poor performance of the Window s version; framerate issues were commonly
mentioned.[81][86] Reviewers noted the story of Elden Ring lacks Martin's writing style. Kyle Orland of Ars
Technica said the game's storytelling is "characteristically sparse and cryptic", and differs from the
expectations of Martin's fans.[76] Chris Carter of Destructoid called the story "low key" but said it is better-
told than those of previous FromSoftware games.[80] Aoife Wilson of Eurogam er said George R. R.
Martin's heavy inclusion in the marketing was "baffling" when his contributions to the overall narrative
were unclear.[72] Mitchell Saltzman did not mind the lack of Martin's style, saying the side-stories rather
than any gr and, ove rarching pl ot kept him "enthralled".[70]

120. Mejia, Ozzie (January 26, 2023). "Elden Ring & Stray lead Game Developers Choice
Awards 2023 nominees" (https://www.shacknews.com/article/133863/gdc-2023-award-nomi
nees). Shacknews. Archived (https://web.archive.org/web/20230127040625/https://www.sha
cknews.com/article/133863/gdc-2023-award-nominees) from the original on January 27,
2023. Retrieved January 27, 2023.
121. Beth Elderkin (March 22, 2023). "'Elden Ring' Wins Game Of The Year At The 2023 Game
Developers Choice Awards" (https://gdconf.com/news/elden-ring-wins-game-year-2023-gam
e-developers-choice-awards). Game Developers Choice Conference. Archived (https://web.
archive.org/web/20230323091858/https://gdconf.com/news/elden-ring-wins-game-year-2023
-game-developers-choice-awards) from the original on March 23, 2023. Retrieved March 23,
2023.
122. "gamescom award 2021: These were the best games of the year" (https://www.gamescom.gl
obal/en/gamescom/gamescom-award/gamescom-award-review-2021). Gamescom.

Tin Pan Alley Award for
Best Music in a GameNominated
The Steam
AwardsJanuary 3, 2023Game of the Year Won
[132]
Best Game You Suck At Won
The Streamer
AwardsMarch 11, 2023Stream Game of the
YearWon[133]
1. Sawyer, Will; Franey, Joel (April 8, 2022). "Where Elden Ring takes place and the story
explained" (https://www.gamesradar.com/elden-ring-where-does-it-take-place-setting-story-l
ore/). gamesradar. Archived (https://web.archive.org/web/20220402212714/https://www.gam
esradar.com/elden-ring-where-does-it-take-place-setting-story-lore/) from the original on April
2, 2022. Retrieved July 26, 2022.
2. Knapp, Mark (June 16, 2021). "Elden Ring: Release Date, Gameplay, and What We Know
So Far" (https://www.ign.com/articles/elden-ring-release-date-news-gameplay-trailer-story).
IGN. Ziff Davis. Archived (https://web.archive.org/web/20220303124310/https://www.ign.co
m/articles/elden-ring-release-date-news-gameplay-trailer-story) from the original on March 3,
2022. Retrieved July 6, 2021.

Elden Ring sold 13.4 million copies worldwide by the end of March 2022,[87] and 20 million by February
2023.[88] It was the best-selling game in several regions between February and March 2022,[89][90] and is
the fastest-selling Bandai Namco game of all time.[91][92] It was the second-bestselling game of 2022 in the
US after Call of Duty: Modern Warfare II,[93] the third-bestselling in Europe,[94] and the tenth-bestselling in
Japan at retail.[95]
Elden Ring won many awards and accolades. It was listed as the
2022' s Game of the Year by several publications, including Ars
Technica,[96] Destructoid,[97] EGM,[98] Eurogam er,[99] Game
Informer,[100] GamesRadar +,[101] GameSpot,[102] IGN,[103] PC
Gamer,[104] and Polygon.[105] It is among the most awarded games
in history with 324 G ame of the Year awards.[106][107]S a l e s
A w a r d s,  Answer the following question - How many awards did Elden Ring Win, and did it win Game of the year award ?.

    Mistral

    1. "Elden Ring" (https://www.metacritic.com/game/pc/elden-ring). Metacritic. Archived (https://web.
archive.org/web/20220402212714/https://www.metacritic.com/game/pc/elden-ring) from the
original on April 2, 2022. Retrieved April 2, 2022.
2. "Elden Ring" (https://www.met

    Llama 3 – 8B

    Elden Ring winning Game of the Year
    at the 23rd Game Developers Choice
    AwardsSome reviewers criticized a number of the game's menu and accessibility systems.[84][85] Reviewers
    complained about the poor performance of the Window s version; framerate issues were commonly
    mentioned.[81][86] Reviewers noted the story of Elden Ring lacks Martin's writing style. Kyle Orland of Ars
    Technica said the game's storytelling is "characteristically sparse and cryptic", and differs from the
    expectations of Martin's fans.[76] Chris Carter

    We can clearly see that Llama 3 – 8B got one part right, however, from the context it couldn’t find that it won 324 awards in total.

    GPT-4

    'Elden Ring won 324 Game of the Year awards. Yes, it did win the Game of the Year award at the 23rd Game Developers Choice Awards.'

    GPT-4 is still far ahead of smaller LLMs, but Llama 3 8B has improved compared to Mistral.

  • Build RAG Application Using Ollama

    In this tutorial, we will build a Retrieval Augmented Generation(RAG) Application using Ollama and Langchain. For the vector store, we will be using Chroma, but you are free to use any vector store of your choice.

    There are 4 key steps to building your RAG application –

    1. Load your documents
    2. Add them to the vector store using the embedding function of your choice.
    3. Define your prompt template.
    4. Deinfe your Retrieval Chatbot using the LLM of your choice.

    In case you want the collab notebook, you can click here.

    First we load the required libraries. 

    # Loading required libraries
    import os

    from langchain.text_splitter import RecursiveCharacterTextSplitter
    from langchain_community.document_loaders import PyPDFLoader
    from langchain_community.vectorstores import Chroma
    from langchain.chains import RetrievalQA
    from langchain.memory import ConversationSummaryMemory
    from langchain_openai import OpenAIEmbeddings
    from langchain.prompts import PromptTemplate
    from langchain.llms import Ollama

    Then comes step 1 which is to load our documents. Here I’ll be using Elden Ring Wiki PDF, you can just visit the Wikipedia page and download it as a PDF file. 

    data_path = "./data/Elden_Ring.pdf"
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=2000,
        chunk_overlap=30,
        length_function=len,)

    documents = PyPDFLoader(data_path).load_and_split(text_splitter=text_splitter)

    The next step is to use an embedding function that will convert our text into embeddings. I prefer using OpenAI embeddings, but you can use any embedding function. Using this embedding function we will add our documents to the Chroma vector database.

    embedding_func = OpenAIEmbeddings(api_key=os.environ.get("OPENAI_API_KEY"))
    vectordb = Chroma.from_documents(documents, embedding=embedding_func)

    Moving on, we have to define a prompt template. I’ll be using the mistral model, so its a very basic prompt template that mistral provides.

    template = """<s>[INST] Given the context - {context} </s>[INST] [INST] Answer the following question - {question}[/INST]"""
    pt = PromptTemplate(
                template=template, input_variables=["context", "question"]
            )

    All that is left to do is to define our memory and Retrieval Chatbot using Ollama as the LLM. 

    rag = RetrievalQA.from_chain_type(
                llm=Ollama(model="mistral"),
                retriever=vectordb.as_retriever(),
                memory=ConversationSummaryMemory(llm = Ollama(model="mistral")),
                chain_type_kwargs={"prompt": pt, "verbose": True},
            )
    rag.invoke("What is Elden Ring ?")
    >>> {'query': 'What is Elden Ring ?',
     'history': '',
     'result': ' Elden Ring is a 2022 action role-playing game developed by FromSoftware. It was published for PlayStation 4, PlayStation 5, Windows, Xbox One, and Xbox Series X/S. In the game, players control a customizable character on a quest to repair the Elden Ring and become the new Elden Lord. The game is set in an open world, presented through a third-person perspective, and includes several types of weapons and magic spells. Players can traverse the six main areas using their steed Torrent and discover linear hidden dungeons and checkpoints that enable fast travel and attribute improvements. Elden Ring features online multiplayer mode for cooperative play or player-versus-player combat. The game was developed with inspirations from Dark Souls series, and contributions from George R.R. Martin on the narrative and Tsukasa Saitoh, Shoi Miyazawa, Tai Tomisawa, Yuka Kitamura, and Yoshimi Kudo for the original soundtrack. Elden Ring received critical acclaim for its open world, gameplay systems, and setting, with some criticism for technical performance. It sold over 20 million copies and a downloadable content expansion, Shadow of the Erdtree, is planned to be released in June 2024.'}

    We see that it was even able to tell us when Shadow of the Erdtree is planned to release for which I’m really excited about. Let me know in the comments if you want to cover anything else.