LangChain Tutorial: Unlock the Potential of LLMs in Question-Answering

In LangChain, index-related chains are the types of chains that allow interaction between LLMs and external data. The external data mainly refers to raw files, such as PDFs, presentations, spreadsheets etc. There is no obvious limit on the volume to data you provide, except costs.

Index-related chains offer the capabilities like summarization, question-answering and text generation. In this tutorial, we will explore different ways of implementing question-answering chain.

Prerequisite¶

We start by setting up the OpenAI API key.

In [36]:

from dotenv import load_dotenv
import os

load_dotenv()
openai_api_key = os.getenv("OPENAI_API_KEY")

To demonstrate different kinds of question-answering chains, we will be using the play Hamlet in the PDF as example. However, you can swap the document loader to load in any supported types of documents.

In [37]:

from langchain.document_loaders import PyPDFLoader
loader = PyPDFLoader("hamlet.pdf")
documents = loader.load()

load_qa_chain¶

This is the most generic type of question-answer chain in LangChain. It literally loads in the full document(s) and answers question based on the information provided.

In [38]:

from langchain.llms import OpenAI
from langchain.chains.question_answering import load_qa_chain

llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
qa_chain = load_qa_chain(llm=llm, chain_type="map_reduce")
query = "Who is Ophelia?"
qa_chain.run(input_documents=documents, question=query)

Out[38]:

" Ophelia is a character in Shakespeare's play Hamlet. She is the daughter of Polonius and sister of Laertes."

Even though there's nothing inherently wrong with this chain, there are obvious limitations with this approach:

• The biggest catch is token limit. It requires more tokens than the typical LLM context window limit to process the whole book, hence why map_reduce chain type is used instead of the default type stuff. Otherwise, it would have triggered token limit error.
• The query operation is very inefficient. Regardless of whichever chain type is used, we are effectively running the whole document, even though in chunks, through LLMs. That's why it takes a long time to generate a response.
• Depending on the LLMs' pricing model, running queries like this can be very expensive.

Therefore, we need to consider a more efficient way to get our answers. This involves the use of embeddings and vector store.

RetrievalQA¶

Under the hood, RetrievalQA chain actually uses load_qa_chain with some distinct features as well. Instead of querying the bulk of text directly, RetrievalQA runs the query on an index containing the embedding values of the text instead.

This is the grand scheme of how it works:

• The raw file will first be split into chunks, to make sure you are not overcrowding LLM with data.
• The text chunks are used to generate embeddings. Typically, more specialised and inexpensive models are used to generate the embedding values.
• Embedding values are then persisted in a special type of database, called vector store. It is optimised for comparing and querying embedding values.
• Instead of feeding the whole documents to LLMs, you can run similarity algorithms to search for similar documents in the vector store, and only feed the top results and the query into LLM as context.

Let's see what that looks like in action.

In [39]:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import FAISS

text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
chunks = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)
knowledge_base = FAISS.from_documents(chunks, embeddings)

In [40]:

from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
retrieval_qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=knowledge_base.as_retriever(), 
    return_source_documents=True
)
query = "Who is Ophelia?"
retrieval_qa_chain({"query": query})

Out[40]:

{'query': 'Who is Ophelia?',
 'result': " Ophelia is a character in Shakespeare's play Hamlet. She is the daughter of Polonius and the love interest of Hamlet.",
 'source_documents': [Document(page_content="OPHELIA\nO heavenly powers, restore him!\nHAMLET\nI have heard of your paintings too, well enough; God\nhas given you one face, and you make yourselves\nanother: you jig, you amble, and you lisp, and\nnick-name God's creatures, and make your wantonness\nyour ignorance. Go to, I'll no more on't; it hath\nmade me mad. I say, we will have no more marriages:\nthose that are married already, all but one, shall\nlive; the rest shall keep as they are. To a\nnunnery, go.\nExit\nOPHELIA\nO, what a noble mind is here o'erthrown!\nThe courtier's, soldier's, scholar's, eye, tongue, sword;\nThe expectancy and rose of the fair state,\nThe glass of fashion and the mould of form,\nThe observed of all observers, quite, quite down!\nAnd I, of ladies most deject and wretched,\nThat suck'd the honey of his music vows,\nNow see that noble and most sovereign reason,\nLike sweet bells jangled, out of tune and harsh;\nThat unmatch'd form and feature of blown youth\nBlasted with ecstasy: O, woe is me,\nTo have seen what I have seen, see what I see!\nRe-enter KING CLAUDIUS and POLONIUS\nKING CLAUDIUS\nLove! his affections do not that way tend;\nNor what he spake, though it lack'd form a little,\nWas not like madness. There's something in his soul,\nO'er which his melancholy sits on brood;\nAnd I do doubt the hatch and the disclose\nWill be some danger: which for to prevent,\nI have in quick determination\nThus set it down: he shall with speed to England,\nFor the demand of our neglected tribute\nHaply the seas and countries different\nWith variable objects shall expel\nThis something-settled matter in his heart,\nWhereon his brains still beating puts him thus\nFrom fashion of himself. What think you on't?\nLORD POLONIUS\nIt shall do well: but yet do I believeHAMLET - Act III\n66", metadata={'source': 'hamlet.pdf', 'page': 65}),
  Document(page_content="OPHELIA\nO, my lord, my lord, I have been so affrighted!\nLORD POLONIUS\nWith what, i' the name of God?\nOPHELIA\nMy lord, as I was sewing in my closet,\nLord Hamlet, with his doublet all unbraced;\nNo hat upon his head; his stockings foul'd,\nUngarter'd, and down-gyved to his ancle;\nPale as his shirt; his knees knocking each other;\nAnd with a look so piteous in purport\nAs if he had been loosed out of hell\nTo speak of horrors,--he comes before me.\nLORD POLONIUS\nMad for thy love?\nOPHELIA\nMy lord, I do not know;\nBut truly, I do fear it.\nLORD POLONIUS\nWhat said he?\nOPHELIA\nHe took me by the wrist and held me hard;\nThen goes he to the length of all his arm;\nAnd, with his other hand thus o'er his brow,\nHe falls to such perusal of my face\nAs he would draw it. Long stay'd he so;\nAt last, a little shaking of mine arm\nAnd thrice his head thus waving up and down,\nHe raised a sigh so piteous and profound\nAs it did seem to shatter all his bulk\nAnd end his being: that done, he lets me go:\nAnd, with his head over his shoulder turn'd,\nHe seem'd to find his way without his eyes;\nFor out o' doors he went without their helps,\nAnd, to the last, bended their light on me.\nLORD POLONIUS\nCome, go with me: I will go seek the king.\nThis is the very ecstasy of love,\nWhose violent property fordoes itself\nAnd leads the will to desperate undertakings\nAs oft as any passion under heaven\nThat does afflict our natures. I am sorry.HAMLET - Act II\n39", metadata={'source': 'hamlet.pdf', 'page': 38}),
  Document(page_content="OPHELIA\nStill better, and worse.\nHAMLET\nSo you must take your husbands. Begin, murderer;\npox, leave thy damnable faces, and begin. Come:\n'the croaking raven doth bellow for revenge.'\nLUCIANUS\nThoughts black, hands apt, drugs fit, and time agreeing;\nConfederate season, else no creature seeing;\nThou mixture rank, of midnight weeds collected,\nWith Hecate's ban thrice blasted, thrice infected,\nThy natural magic and dire property,\nOn wholesome life usurp immediately.\nPours the poison into the sleeper's ears\nHAMLET\nHe poisons him i' the garden for's estate. His\nname's Gonzago: the story is extant, and writ in\nchoice Italian: you shall see anon how the murderer\ngets the love of Gonzago's wife.\nOPHELIA\nThe king rises.\nHAMLET\nWhat, frighted with false fire!\nQUEEN GERTRUDE\nHow fares my lord?\nLORD POLONIUS\nGive o'er the play.\nKING CLAUDIUS\nGive me some light: away!\nAll\nLights, lights, lights!\nExeunt all but HAMLET and HORATIO\nHAMLET\nWhy, let the stricken deer go weep,\nThe hart ungalled play;\nFor some must watch, while some must sleep:\nSo runs the world away.\nWould not this, sir, and a forest of feathers-- if\nthe rest of my fortunes turn Turk with me--with twoHAMLET - Act III\n76", metadata={'source': 'hamlet.pdf', 'page': 75}),
  Document(page_content="Let her come in.\nLAERTES\nHow now! what noise is that?\nRe-enter OPHELIA\nO heat, dry up my brains! tears seven times salt,\nBurn out the sense and virtue of mine eye!\nBy heaven, thy madness shall be paid by weight,\nTill our scale turn the beam. O rose of May!\nDear maid, kind sister, sweet Ophelia!\nO heavens! is't possible, a young maid's wits\nShould be as moral as an old man's life?\nNature is fine in love, and where 'tis fine,\nIt sends some precious instance of itself\nAfter the thing it loves.\nOPHELIA\nSings\nThey bore him barefaced on the bier;\nHey non nonny, nonny, hey nonny;\nAnd in his grave rain'd many a tear:--\nFare you well, my dove!\nLAERTES\nHadst thou thy wits, and didst persuade revenge,\nIt could not move thus.\nOPHELIA\nSings\nYou must sing a-down a-down,\nAn you call him a-down-a.\nO, how the wheel becomes it! It is the false\nsteward, that stole his master's daughter.\nLAERTES\nThis nothing's more than matter.\nOPHELIA\nThere's rosemary, that's for remembrance; pray,\nlove, remember: and there is pansies. that's for thoughts.\nLAERTES\nA document in madness, thoughts and remembrance fitted.\nOPHELIA\nThere's fennel for you, and columbines: there's rue\nfor you; and here's some for me: we may call it\nherb-grace o' Sundays: O you must wear your rue with\na difference. There's a daisy: I would give youHAMLET - Act IV\n106", metadata={'source': 'hamlet.pdf', 'page': 105})]}

There is twist to RetrievalQA chain. As index is a basic component in LangChain, it doesn't rely on chain to function. In fact, queries can be run directly with VectorstoreIndexCreator once the embedding values are ready. The index provides a wrapper around RetrievalQA.

That said, it's not entirely clear to me why this wrapper was created, and it doesn't seem to fit into the current workflow.

In [58]:

from langchain.indexes import VectorstoreIndexCreator

llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
vectorstore_index = VectorstoreIndexCreator(
    vectorstore_cls=FAISS, 
    embedding=embeddings,
    text_splitter=text_splitter
).from_documents(documents)
query = "Who is Ophelia?"
vectorstore_index.query(llm=llm, question=query)

Out[58]:

" Ophelia is a character in Shakespeare's play Hamlet. She is the daughter of Polonius and the love interest of Hamlet."

ConversationalRetrievalChain¶

So far, you should have a good understanding of how to query index-related chains. But to make the experience more real-life like, the query needs to happen in a conversational style. To make the question-answering chain more fluent, it needs to have a memory. This is what ConversationalRetrievalChain is created to do, it is basically a RetrievalQA chain topping up with a memory.

In [54]:

from langchain.llms import OpenAI
from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationalRetrievalChain

llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
conversational_retrieval_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    chain_type="stuff",
    retriever=knowledge_base.as_retriever(), 
    memory=memory
)
query = "Who is Ophelia?"
result = conversational_retrieval_chain({"question": query})
result["answer"]

Out[54]:

" Ophelia is a character in Shakespeare's play Hamlet. She is the daughter of Polonius and the love interest of Hamlet."

In [55]:

query = "Is she the daughter of King Hamlet?"
result = conversational_retrieval_chain({"question": query})
result["answer"]

Out[55]:

' No, Ophelia is the daughter of Lord Polonius.'

Now, we've been firing away with two questions. You can see from the second question, it requires the LLM to infer she before giving a response. This is done by preserving the previous questions and answers in an object called ConversationBufferMemory, and passed to LLMs as part of context.

In [56]:

memory

Out[56]:

ConversationBufferMemory(chat_memory=ChatMessageHistory(messages=[HumanMessage(content='Who is Ophelia?', additional_kwargs={}, example=False), AIMessage(content=" Ophelia is a character in Shakespeare's play Hamlet. She is the daughter of Polonius and the love interest of Hamlet.", additional_kwargs={}, example=False), HumanMessage(content='Is she the daughter of King Hamlet?', additional_kwargs={}, example=False), AIMessage(content=' No, Ophelia is the daughter of Lord Polonius.', additional_kwargs={}, example=False)]), output_key=None, input_key=None, return_messages=True, human_prefix='Human', ai_prefix='AI', memory_key='chat_history')

Equally, you can also manage a memory manually, and this is what it looks like.

In [44]:

conversational_retrieval_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    chain_type="stuff",
    retriever=knowledge_base.as_retriever()
)
chat_history = []
query = "Who is Ophelia?"
result = conversational_retrieval_chain({"question": query, "chat_history": chat_history})
result["answer"]

Out[44]:

" Ophelia is a character in Shakespeare's play Hamlet. She is the daughter of Polonius and the love interest of Hamlet."

In [45]:

chat_history = [(query, result["answer"])]
query = "Is she the daughter of King Hamlet?"
result = conversational_retrieval_chain({"question": query, "chat_history": chat_history})
result["answer"]

Out[45]:

' No, Ophelia is the daughter of Lord Polonius.'

In [46]:

chat_history

Out[46]:

[('Who is Ophelia?',
  " Ophelia is a character in Shakespeare's play Hamlet. She is the daughter of Polonius and the love interest of Hamlet.")]

GraphQAChain¶

To complete the topic of question-answer chain. There is a more niche chain, called GraphQAChain, it is implemented with graph instead of vector values.

In LangChain 101, we have discussed the fundamental LangChain components, including Indexes. The indexing process does not only mean the process of vectorisation, it can also be used to build a graph index.

As graph index works best for small pieces of text, we will only be using one chunk of the whole document to demonstrate how this chain works.

In [47]:

documents[2].page_content

Out[47]:

"Dramatis Personae\nCLAUDIUS, king of Denmark.\nHAMLET, son to the late, and nephew to the present king.\nPOLONIUS, lord chamberlain.\nHORATIO, friend to Hamlet.\nLAERTES, son to Polonius.\nLUCIANUS, nephew to the king.\nVOLTIMAND\nCORNELIUS\nROSENCRANTZ\nGUILDENSTERN\nOSRIC\ncourtiers.\nA Gentleman\nA Priest.\nMARCELLUS\nBERNARDO\nofficers.\nFRANCISCO, a soldier.\nREYNALDO, servant to Polonius.\nPlayers.\nTwo Clowns, grave-diggers.\nFORTINBRAS, prince of Norway.\nA Captain.\nEnglish Ambassadors.\nGERTRUDE, queen of Denmark, and mother to Hamlet.\nOPHELIA, daughter to Polonius.\nLords, Ladies, Officers, Soldiers, Sailors, Messengers, and other Attendants.\nGhost of Hamlet's Father."

We will first build a graph of relationships in triples using GraphIndexCreator.

In [48]:

from langchain.llms import OpenAI
from langchain.indexes import GraphIndexCreator

llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
graph_index = GraphIndexCreator(llm=llm)
graph = graph_index.from_text(documents[2].page_content)

In [49]:

graph.get_triples()

Out[49]:

[('Claudius', 'king of Denmark', 'is'),
 ('Hamlet', 'son to the late', 'is'),
 ('Hamlet', 'nephew to the present king', 'is'),
 ('Polonius', 'lord chamberlain', 'is'),
 ('Horatio', 'friend to Hamlet', 'is'),
 ('Laertes', 'son to Polonius', 'is'),
 ('Lucianus', 'nephew to the king', 'is'),
 ('Voltimand', 'courtier', 'is'),
 ('Cornelius', 'courtier', 'is'),
 ('Rosencrantz', 'courtier', 'is'),
 ('Guildenstern', 'courtier', 'is'),
 ('Osric', 'courtier', 'is'),
 ('Gentleman', 'courtier', 'is'),
 ('Priest', 'courtier', 'is'),
 ('Marcellus', 'officer', 'is'),
 ('Bernardo', 'officer', 'is'),
 ('Francisco', 'soldier', 'is'),
 ('Reynaldo', 'servant to Polonius', 'is'),
 ('Players', 'courtiers', 'are'),
 ('Two Clowns', 'grave-diggers', 'are')]

Provide the graph to GraphQAChain for queries.

In [50]:

from langchain.chains import GraphQAChain

graph_qa_chain = GraphQAChain.from_llm(llm=llm, graph=graph, verbose=True)
query = "Who is Hamlet to Claudius?"
graph_qa_chain.run(query)

> Entering new GraphQAChain chain...
Entities Extracted:
 Hamlet, Claudius
Full Context:
Hamlet is son to the late
Hamlet is nephew to the present kingClaudius is king of Denmark

> Finished chain.

Out[50]:

" Hamlet is Claudius's nephew."

You may also be interested to know, there's no twist like VectorstoreIndexCreator this time, and the only way to query the graph index is via the chain itself.