LangChain Tutorial: Chain Types

Once you get more familiar with different types of chains, you will notice there is a configuration called chain types at instantiation. So what does that mean? What does it do? When should you consider using a particular chain type? Don't worry, you will find answers to all of those questions in this article.

In short, this is the configuration that dictates which prompt technique is used to achieve the task. The difference in the results produced may be less clear, however, it could impact the performance of chain, and determine how effective the LLMs are being used.

I will use summarization chain as example to explain further. For the patient, the difference are listed in the table below for your reference.

Stuff	Map Reduce	Refine	Rerank
Generate text based on a single prompt.	Uses a divide-and-conquer approach to break down a large task into smaller tasks.	Uses a feedback loop to improve the quality of generated text.	Uses a ranking algorithm to reorder generated text based on its quality.
Can generate creative text that is both original and interesting.	Can be used to efficiently parallelize large tasks.	Can improve the quality of generated text by making it more specific, detailed, and relevant to the prompt.	Can improve the ranking of generated text by reordering it based on its quality.
Slow for large prompts	Difficult to implement for large tasks	Time-consuming to train the feedback loop	Sensitive to the quality of the ranking algorithm
Poems, code, scripts, music, email, letters	Image, video, NLP processing	Chatbots, machine translation, text summarization	Search results, product recommendations, news articles

Prerequisite¶

Load in your OpenAI API key.

In [1]:

from dotenv import load_dotenv
import os

load_dotenv()
openai_api_key = os.getenv("OPENAI_API_KEY")

I will be using the famous I have a dream speech by Dr. Martin Luther King to showcase the difference and highlight the steps each chain type takes to complete the task.

In [2]:

from langchain.document_loaders import PyPDFLoader

loader = PyPDFLoader("i_have_a_dream.pdf")
documents = loader.load()

In order to see the intermediate steps clearly, I chose this well-known speech that has roughly around 1674 words, 9494 characters in total.

In [3]:

num_of_chars = 0
num_of_words = 0
for document in documents:
    num_of_chars += len(document.page_content)
    num_of_words += len(document.page_content.split())

print(f"number of pages: {len(documents)}")
print(f"number of characters: {num_of_chars}")
print(f"number of words: {num_of_words}")

number of pages: 3
number of characters: 9494
number of words: 1674

Stuff Chain¶

Stuff chain type is the default and probably the most commonly used type. It is good at generating content that is both original and interesting. The biggest catch is it does everything in one single prompt.

You can quickly that catch can quickly become very problematic, especially when we need to pass a huge amount of text as context to LLMs, which will most likely exceed the token limit.

In fact, you can see clearly the error being thrown by this naive approach.

In [4]:

from langchain.llms import OpenAI
from langchain.chains.summarize import load_summarize_chain

llm = OpenAI(temperature=0, openai_api_key=openai_api_key)
stuff_chain = load_summarize_chain(llm=llm, chain_type="stuff", verbose=True)

In [5]:

stuff_chain.run(documents)

> Entering new StuffDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
Write a concise summary of the following:


"I HAVE A DREAM  
Martin Luther King, Jr. 
 
I am happy to join with you today in what will go down in history as the greatest demonstration  for freedom in the 
history of our nation. 
 Five score years ago, a great American,  in whose symbolic shadow we stand today, signed the Emancipation  
Proclamation.  This 
momentous  decree came as a great beacon light of hope to millions of Negro slaves who had been 
seared in the flames of withering  injustice. It came as a joyous daybreak  to end the long night of their captivity. 
 But one hundred years later, the Negro still is not free. One
 hundred years later, the life of the Negro is still sadly 
crippled by the manacles  of segregation  and the chains of discrimination.  One hundred years later, the Negro lives on a 
lonely island of poverty in the midst of a vast ocean of material prosperity.  One hundred years later, the Negro is still 
languished  in the corners of American  society and finds himself an exile in his own land. And so we've come here today 
to dramatize  a shameful  condition.  
 In a sense we've come to our nation's capital to cash a check. When the architects  of our republic wrote the
 magnificent  
words of the Constitution  and the Declaration  of Independence,  they were signing a promissory  note to which every 
American  was to fall heir. This note was a promise that all men, yes, black men as well as white men, would be 
guaranteed  the "unalienable  Rights" of "Life, Liberty and the pursuit of Happiness."  It is obvious today that America has 
defaulted  on this promissory  note, insofar as her citizens of color are concerned.  Instead of honoring this sacred 
obligation,  America has given the Negro people a bad check, a check which has come back marked "insufficient  funds." 
 
But we refuse to believe that the bank of justice is bankrupt.  We refuse to believe that there are insufficient  funds in the 
great vaults of opportunity  of this nation. And so, we've come to cash this check, a check that will give us upon demand 
the riches of freedom and the security of justice. 
 
We have also come to this hallowed spot to remind America of the fierce urgency of Now. This is no time to engage in 
the luxury of cooling off or to take the tranquilizing  drug of gradualism.  Now is the time to make real the promises of 
democracy.  Now is the time to rise from the dark and desolate valley of segregation  to the sunlit path of racial justice. 
Now is the time to lift our nation from the quicksands  of racial injustice to the solid rock of brotherhood.  Now is the time 
to make justice a reality for all of God's children. 
 
It would be fatal for the nation to overlook the urgency of the moment. This sweltering  summer of the Negro's 
legitimate  discontent  will not pass until there is an invigorating  autumn of freedom and equality. Nineteen  sixty‐three is 
not an end, but a beginning.  And those who hope that the Negro needed to blow off steam and will now be content will 
have a rude awakening  if the nation returns to business as usual. And there will be neither rest nor tranquility  in America 
until the Negro is granted his citizenship  rights. The whirlwinds  of revolt will continue to shake the foundations  of our 
nation until the bright day of justice emerges.  
 But there is something  that I must say to my people, who stand on the warm threshold  which leads into the palace of 
justice: In the process of gaining our rightful place, we 
must not be guilty of wrongful deeds. Let us not seek to satisfy 
our thirst for freedom by drinking from the cup of bitterness  and hatred. We must forever conduct our struggle on the 
high plane of dignity and discipline.  We must not allow our creative protest to degenerate  into physical violence. Again 
and again, we must rise to the majestic heights of meeting physical force with soul force. 
 The marvelous  new militancy which has engulfed the Negro community  must not lead us to a distrust of all white 
people, for many of our white brothers,  as evidenced  by their presence 
here today, have come to realize that their 
destiny is tied up with our destiny. And they have come to realize that their freedom is inextricably  bound to our 
freedom.  
 We cannot walk alone. 
 

And as we walk, we must make the pledge that we shall always march ahead. 
 We cannot turn back. 
 
There are those who are asking the devotees of civil rights, "When will you be satisfied?"  We can never be satisfied as 
long as the Negro is the victim of the 
unspeakable  horrors of police brutality. We can never be satisfied as long as our 
bodies, heavy with the fatigue of travel, cannot gain lodging in the motels of the highways  and the hotels of the cities. 
We cannot be satisfied as long as the negro's basic mobility is from a smaller ghetto to a larger one. We can never be 
satisfied as long as our children are stripped of their self‐hood and robbed of their dignity by signs stating: "For Whites 
Only." We cannot be satisfied as long as a Negro in Mississippi  cannot vote and a Negro in New York believes he has 
nothing for which to vote. No, no, we are not satisfied, and we will not be satisfied until "justice rolls down like waters, 
and righteousness  like a mighty stream."¹  
 
I am not unmindful  that some of you have come here out of great trials and tribulations.  Some of you have come fresh 
from narrow jail cells. And some of you have come from areas where your quest ‐‐ quest for freedom left you battered 
by the storms of persecution  and staggered  by the winds of police brutality. You have been the veterans of creative 
suffering.  Continue to work with the faith that unearned  suffering is redemptive.  Go back to Mississippi,  go back to 
Alabama,  go back to South Carolina, go back to Georgia, go back to Louisiana,  go back to the slums and ghettos of our 
northern cities, knowing that somehow  this situation can and will be changed. 
 
Let us not wallow in the valley of despair, I say to you today, my friends. 
 And so even though we face the difficulties  of today and tomorrow,  I still have a dream. It is a dream deeply rooted in 
the American  dream. 
 I have a dream that one day 
this nation will rise up and live out the true meaning of its creed: "We hold these truths to 
be self‐evident, that all men are created equal." 
 
I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners 
will be able to sit down together at the table of brotherhood.  
 I have a dream that one day even the state of Mississippi,  a state sweltering  with the heat of injustice, sweltering  with 
the heat of oppression,  will be transformed  into an oasis of freedom and justice.
 
 I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their 
skin but by the content of their character.  
 
I have a dream today! 
 I have a dream that one day, down in Alabama,
 with its vicious racists, with its governor having his lips dripping with the 
words of "interposition"  and "nullification"  ‐‐ one day right there in Alabama little black boys and black girls will be able 
to join hands with little white boys and white girls as sisters and brothers.  
 
I have a dream today! 
 I have a dream that one day every valley shall be exalted, and every hill and mountain  shall be made low, the rough 
places will be made plain, and the crooked places will be made straight; "and the glory of the Lord shall be revealed and 
all flesh 
shall see it together."2  
 This is our hope, and this is the faith that I go back to the South with. 
 With this faith, we will be able to hew out of the mountain  of despair a stone of hope. With this faith, we will be able to 
transform  the jangling
 discords of our nation into a beautiful symphony  of brotherhood.  With this faith, we will be able 

to work together,  to pray together,  to struggle together,  to go to jail together,  to stand up for freedom together,  
knowing that we will be free one day. 
 
And this will be the day ‐‐ this will be the day when all of God's children will be able to sing with new meaning:  
 My country 'tis of thee, sweet land of liberty, of thee I sing. 
 Land where my fathers died, land of the Pilgrim's pride, 
 
From every mountainside,  let freedom ring! 
 And if America is to be a great nation, this must become true. 
 
And so let freedom ring from
 the prodigious  hilltops of New Hampshire.  
 Let freedom ring from the mighty mountains  of New York. 
 Let freedom ring from the heightening  Alleghenies  of Pennsylvania.  
 
Let freedom ring from the snow‐capped Rockies of Colorado.  
 Let freedom ring from the curvaceous  slopes of California.  
 
But not only that: 
 Let 
freedom ring from Stone Mountain  of Georgia. 
 
Let freedom ring from Lookout Mountain  of Tennessee.  
 Let freedom ring from every hill and molehill of Mississippi.  
 From every mountainside,  let freedom ring. 
 
And when this happens, when we allow freedom ring, when we let it ring from every village and every
 hamlet, from 
every state and every city, we will be able to speed up that day when all of God's children, black men and white men, 
Jews and Gentiles, Protestants  and Catholics,  will be able to join hands and sing in the words of the old Negro spiritual: 
 
                 Free at last! Free at last! 
                  Thank God Almighty,  we are free at last!3 "


CONCISE SUMMARY:

---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)
Cell In[5], line 1
----> 1 stuff_chain.run(documents)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/base.py:236, in Chain.run(self, callbacks, *args, **kwargs)
    234     if len(args) != 1:
    235         raise ValueError("`run` supports only one positional argument.")
--> 236     return self(args[0], callbacks=callbacks)[self.output_keys[0]]
    238 if kwargs and not args:
    239     return self(kwargs, callbacks=callbacks)[self.output_keys[0]]

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/base.py:140, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
    138 except (KeyboardInterrupt, Exception) as e:
    139     run_manager.on_chain_error(e)
--> 140     raise e
    141 run_manager.on_chain_end(outputs)
    142 return self.prep_outputs(inputs, outputs, return_only_outputs)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/base.py:134, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
    128 run_manager = callback_manager.on_chain_start(
    129     {"name": self.__class__.__name__},
    130     inputs,
    131 )
    132 try:
    133     outputs = (
--> 134         self._call(inputs, run_manager=run_manager)
    135         if new_arg_supported
    136         else self._call(inputs)
    137     )
    138 except (KeyboardInterrupt, Exception) as e:
    139     run_manager.on_chain_error(e)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/combine_documents/base.py:84, in BaseCombineDocumentsChain._call(self, inputs, run_manager)
     82 # Other keys are assumed to be needed for LLM prediction
     83 other_keys = {k: v for k, v in inputs.items() if k != self.input_key}
---> 84 output, extra_return_dict = self.combine_docs(
     85     docs, callbacks=_run_manager.get_child(), **other_keys
     86 )
     87 extra_return_dict[self.output_key] = output
     88 return extra_return_dict

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/combine_documents/stuff.py:87, in StuffDocumentsChain.combine_docs(self, docs, callbacks, **kwargs)
     85 inputs = self._get_inputs(docs, **kwargs)
     86 # Call predict on the LLM.
---> 87 return self.llm_chain.predict(callbacks=callbacks, **inputs), {}

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/llm.py:213, in LLMChain.predict(self, callbacks, **kwargs)
    198 def predict(self, callbacks: Callbacks = None, **kwargs: Any) -> str:
    199     """Format prompt with kwargs and pass to LLM.
    200 
    201     Args:
   (...)
    211             completion = llm.predict(adjective="funny")
    212     """
--> 213     return self(kwargs, callbacks=callbacks)[self.output_key]

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/base.py:140, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
    138 except (KeyboardInterrupt, Exception) as e:
    139     run_manager.on_chain_error(e)
--> 140     raise e
    141 run_manager.on_chain_end(outputs)
    142 return self.prep_outputs(inputs, outputs, return_only_outputs)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/base.py:134, in Chain.__call__(self, inputs, return_only_outputs, callbacks)
    128 run_manager = callback_manager.on_chain_start(
    129     {"name": self.__class__.__name__},
    130     inputs,
    131 )
    132 try:
    133     outputs = (
--> 134         self._call(inputs, run_manager=run_manager)
    135         if new_arg_supported
    136         else self._call(inputs)
    137     )
    138 except (KeyboardInterrupt, Exception) as e:
    139     run_manager.on_chain_error(e)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/llm.py:69, in LLMChain._call(self, inputs, run_manager)
     64 def _call(
     65     self,
     66     inputs: Dict[str, Any],
     67     run_manager: Optional[CallbackManagerForChainRun] = None,
     68 ) -> Dict[str, str]:
---> 69     response = self.generate([inputs], run_manager=run_manager)
     70     return self.create_outputs(response)[0]

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/chains/llm.py:79, in LLMChain.generate(self, input_list, run_manager)
     77 """Generate LLM result from inputs."""
     78 prompts, stop = self.prep_prompts(input_list, run_manager=run_manager)
---> 79 return self.llm.generate_prompt(
     80     prompts, stop, callbacks=run_manager.get_child() if run_manager else None
     81 )

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/llms/base.py:134, in BaseLLM.generate_prompt(self, prompts, stop, callbacks)
    127 def generate_prompt(
    128     self,
    129     prompts: List[PromptValue],
    130     stop: Optional[List[str]] = None,
    131     callbacks: Callbacks = None,
    132 ) -> LLMResult:
    133     prompt_strings = [p.to_string() for p in prompts]
--> 134     return self.generate(prompt_strings, stop=stop, callbacks=callbacks)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/llms/base.py:191, in BaseLLM.generate(self, prompts, stop, callbacks)
    189 except (KeyboardInterrupt, Exception) as e:
    190     run_manager.on_llm_error(e)
--> 191     raise e
    192 run_manager.on_llm_end(output)
    193 return output

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/llms/base.py:185, in BaseLLM.generate(self, prompts, stop, callbacks)
    180 run_manager = callback_manager.on_llm_start(
    181     {"name": self.__class__.__name__}, prompts, invocation_params=params
    182 )
    183 try:
    184     output = (
--> 185         self._generate(prompts, stop=stop, run_manager=run_manager)
    186         if new_arg_supported
    187         else self._generate(prompts, stop=stop)
    188     )
    189 except (KeyboardInterrupt, Exception) as e:
    190     run_manager.on_llm_error(e)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/llms/openai.py:314, in BaseOpenAI._generate(self, prompts, stop, run_manager)
    312     choices.extend(response["choices"])
    313 else:
--> 314     response = completion_with_retry(self, prompt=_prompts, **params)
    315     choices.extend(response["choices"])
    316 if not self.streaming:
    317     # Can't update token usage if streaming

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/llms/openai.py:106, in completion_with_retry(llm, **kwargs)
    102 @retry_decorator
    103 def _completion_with_retry(**kwargs: Any) -> Any:
    104     return llm.client.create(**kwargs)
--> 106 return _completion_with_retry(**kwargs)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/tenacity/__init__.py:289, in BaseRetrying.wraps.<locals>.wrapped_f(*args, **kw)
    287 @functools.wraps(f)
    288 def wrapped_f(*args: t.Any, **kw: t.Any) -> t.Any:
--> 289     return self(f, *args, **kw)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/tenacity/__init__.py:379, in Retrying.__call__(self, fn, *args, **kwargs)
    377 retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)
    378 while True:
--> 379     do = self.iter(retry_state=retry_state)
    380     if isinstance(do, DoAttempt):
    381         try:

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/tenacity/__init__.py:314, in BaseRetrying.iter(self, retry_state)
    312 is_explicit_retry = fut.failed and isinstance(fut.exception(), TryAgain)
    313 if not (is_explicit_retry or self.retry(retry_state)):
--> 314     return fut.result()
    316 if self.after is not None:
    317     self.after(retry_state)

File ~/.pyenv/versions/3.10.11/lib/python3.10/concurrent/futures/_base.py:451, in Future.result(self, timeout)
    449     raise CancelledError()
    450 elif self._state == FINISHED:
--> 451     return self.__get_result()
    453 self._condition.wait(timeout)
    455 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File ~/.pyenv/versions/3.10.11/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
    401 if self._exception:
    402     try:
--> 403         raise self._exception
    404     finally:
    405         # Break a reference cycle with the exception in self._exception
    406         self = None

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/tenacity/__init__.py:382, in Retrying.__call__(self, fn, *args, **kwargs)
    380 if isinstance(do, DoAttempt):
    381     try:
--> 382         result = fn(*args, **kwargs)
    383     except BaseException:  # noqa: B902
    384         retry_state.set_exception(sys.exc_info())  # type: ignore[arg-type]

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/langchain/llms/openai.py:104, in completion_with_retry.<locals>._completion_with_retry(**kwargs)
    102 @retry_decorator
    103 def _completion_with_retry(**kwargs: Any) -> Any:
--> 104     return llm.client.create(**kwargs)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/openai/api_resources/completion.py:25, in Completion.create(cls, *args, **kwargs)
     23 while True:
     24     try:
---> 25         return super().create(*args, **kwargs)
     26     except TryAgain as e:
     27         if timeout is not None and time.time() > start + timeout:

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py:153, in EngineAPIResource.create(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)
    127 @classmethod
    128 def create(
    129     cls,
   (...)
    136     **params,
    137 ):
    138     (
    139         deployment_id,
    140         engine,
   (...)
    150         api_key, api_base, api_type, api_version, organization, **params
    151     )
--> 153     response, _, api_key = requestor.request(
    154         "post",
    155         url,
    156         params=params,
    157         headers=headers,
    158         stream=stream,
    159         request_id=request_id,
    160         request_timeout=request_timeout,
    161     )
    163     if stream:
    164         # must be an iterator
    165         assert not isinstance(response, OpenAIResponse)

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/openai/api_requestor.py:226, in APIRequestor.request(self, method, url, params, headers, files, stream, request_id, request_timeout)
    205 def request(
    206     self,
    207     method,
   (...)
    214     request_timeout: Optional[Union[float, Tuple[float, float]]] = None,
    215 ) -> Tuple[Union[OpenAIResponse, Iterator[OpenAIResponse]], bool, str]:
    216     result = self.request_raw(
    217         method.lower(),
    218         url,
   (...)
    224         request_timeout=request_timeout,
    225     )
--> 226     resp, got_stream = self._interpret_response(result, stream)
    227     return resp, got_stream, self.api_key

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/openai/api_requestor.py:620, in APIRequestor._interpret_response(self, result, stream)
    612     return (
    613         self._interpret_response_line(
    614             line, result.status_code, result.headers, stream=True
    615         )
    616         for line in parse_stream(result.iter_lines())
    617     ), True
    618 else:
    619     return (
--> 620         self._interpret_response_line(
    621             result.content.decode("utf-8"),
    622             result.status_code,
    623             result.headers,
    624             stream=False,
    625         ),
    626         False,
    627     )

File ~/.pyenv/versions/3.10.11/lib/python3.10/site-packages/openai/api_requestor.py:683, in APIRequestor._interpret_response_line(self, rbody, rcode, rheaders, stream)
    681 stream_error = stream and "error" in resp.data
    682 if stream_error or not 200 <= rcode < 300:
--> 683     raise self.handle_error_response(
    684         rbody, rcode, resp.data, rheaders, stream_error=stream_error
    685     )
    686 return resp

InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 4514 tokens (4258 in your prompt; 256 for the completion). Please reduce your prompt; or completion length.

I understand that's an disappointing start, but more than enough to prove the point. Regardless of which, let's at least make sure summarization chain works so that we can continue the rest of the experiment. This time I will only supply the first page of the document to the LLM.

In [6]:

stuff_chain.run([documents[0]])

> Entering new StuffDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
Write a concise summary of the following:


"I HAVE A DREAM  
Martin Luther King, Jr. 
 
I am happy to join with you today in what will go down in history as the greatest demonstration  for freedom in the 
history of our nation. 
 Five score years ago, a great American,  in whose symbolic shadow we stand today, signed the Emancipation  
Proclamation.  This 
momentous  decree came as a great beacon light of hope to millions of Negro slaves who had been 
seared in the flames of withering  injustice. It came as a joyous daybreak  to end the long night of their captivity. 
 But one hundred years later, the Negro still is not free. One
 hundred years later, the life of the Negro is still sadly 
crippled by the manacles  of segregation  and the chains of discrimination.  One hundred years later, the Negro lives on a 
lonely island of poverty in the midst of a vast ocean of material prosperity.  One hundred years later, the Negro is still 
languished  in the corners of American  society and finds himself an exile in his own land. And so we've come here today 
to dramatize  a shameful  condition.  
 In a sense we've come to our nation's capital to cash a check. When the architects  of our republic wrote the
 magnificent  
words of the Constitution  and the Declaration  of Independence,  they were signing a promissory  note to which every 
American  was to fall heir. This note was a promise that all men, yes, black men as well as white men, would be 
guaranteed  the "unalienable  Rights" of "Life, Liberty and the pursuit of Happiness."  It is obvious today that America has 
defaulted  on this promissory  note, insofar as her citizens of color are concerned.  Instead of honoring this sacred 
obligation,  America has given the Negro people a bad check, a check which has come back marked "insufficient  funds." 
 
But we refuse to believe that the bank of justice is bankrupt.  We refuse to believe that there are insufficient  funds in the 
great vaults of opportunity  of this nation. And so, we've come to cash this check, a check that will give us upon demand 
the riches of freedom and the security of justice. 
 
We have also come to this hallowed spot to remind America of the fierce urgency of Now. This is no time to engage in 
the luxury of cooling off or to take the tranquilizing  drug of gradualism.  Now is the time to make real the promises of 
democracy.  Now is the time to rise from the dark and desolate valley of segregation  to the sunlit path of racial justice. 
Now is the time to lift our nation from the quicksands  of racial injustice to the solid rock of brotherhood.  Now is the time 
to make justice a reality for all of God's children. 
 
It would be fatal for the nation to overlook the urgency of the moment. This sweltering  summer of the Negro's 
legitimate  discontent  will not pass until there is an invigorating  autumn of freedom and equality. Nineteen  sixty‐three is 
not an end, but a beginning.  And those who hope that the Negro needed to blow off steam and will now be content will 
have a rude awakening  if the nation returns to business as usual. And there will be neither rest nor tranquility  in America 
until the Negro is granted his citizenship  rights. The whirlwinds  of revolt will continue to shake the foundations  of our 
nation until the bright day of justice emerges.  
 But there is something  that I must say to my people, who stand on the warm threshold  which leads into the palace of 
justice: In the process of gaining our rightful place, we 
must not be guilty of wrongful deeds. Let us not seek to satisfy 
our thirst for freedom by drinking from the cup of bitterness  and hatred. We must forever conduct our struggle on the 
high plane of dignity and discipline.  We must not allow our creative protest to degenerate  into physical violence. Again 
and again, we must rise to the majestic heights of meeting physical force with soul force. 
 The marvelous  new militancy which has engulfed the Negro community  must not lead us to a distrust of all white 
people, for many of our white brothers,  as evidenced  by their presence 
here today, have come to realize that their 
destiny is tied up with our destiny. And they have come to realize that their freedom is inextricably  bound to our 
freedom.  
 We cannot walk alone. 
 "


CONCISE SUMMARY:

> Finished chain.

> Finished chain.

Out[6]:

' Martin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people.'

You can see the summarization chain works as sweet as a nut, but how are we gonna get over the token limit problem? Wait for it, here it comes!

Map Reduce Chain¶

The first type of prompting technique that can help with the token limit is called Map Reduce. For those who are familiar with the concept of map reduce, it does exactly what it says on the tin, which is to break down a large task into smaller, more manageable tasks that can be executed in parallel.

The key advantages of this approach is it can make the operation more efficient by parallelizing tasks, and actually allows us to work with large amount text.

Let's take a look at the traces.

In [7]:

map_reduce_chain = load_summarize_chain(llm=llm, chain_type="map_reduce", verbose=True, return_intermediate_steps=True)
result = map_reduce_chain(documents)

> Entering new MapReduceDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
Write a concise summary of the following:


"I HAVE A DREAM  
Martin Luther King, Jr. 
 
I am happy to join with you today in what will go down in history as the greatest demonstration  for freedom in the 
history of our nation. 
 Five score years ago, a great American,  in whose symbolic shadow we stand today, signed the Emancipation  
Proclamation.  This 
momentous  decree came as a great beacon light of hope to millions of Negro slaves who had been 
seared in the flames of withering  injustice. It came as a joyous daybreak  to end the long night of their captivity. 
 But one hundred years later, the Negro still is not free. One
 hundred years later, the life of the Negro is still sadly 
crippled by the manacles  of segregation  and the chains of discrimination.  One hundred years later, the Negro lives on a 
lonely island of poverty in the midst of a vast ocean of material prosperity.  One hundred years later, the Negro is still 
languished  in the corners of American  society and finds himself an exile in his own land. And so we've come here today 
to dramatize  a shameful  condition.  
 In a sense we've come to our nation's capital to cash a check. When the architects  of our republic wrote the
 magnificent  
words of the Constitution  and the Declaration  of Independence,  they were signing a promissory  note to which every 
American  was to fall heir. This note was a promise that all men, yes, black men as well as white men, would be 
guaranteed  the "unalienable  Rights" of "Life, Liberty and the pursuit of Happiness."  It is obvious today that America has 
defaulted  on this promissory  note, insofar as her citizens of color are concerned.  Instead of honoring this sacred 
obligation,  America has given the Negro people a bad check, a check which has come back marked "insufficient  funds." 
 
But we refuse to believe that the bank of justice is bankrupt.  We refuse to believe that there are insufficient  funds in the 
great vaults of opportunity  of this nation. And so, we've come to cash this check, a check that will give us upon demand 
the riches of freedom and the security of justice. 
 
We have also come to this hallowed spot to remind America of the fierce urgency of Now. This is no time to engage in 
the luxury of cooling off or to take the tranquilizing  drug of gradualism.  Now is the time to make real the promises of 
democracy.  Now is the time to rise from the dark and desolate valley of segregation  to the sunlit path of racial justice. 
Now is the time to lift our nation from the quicksands  of racial injustice to the solid rock of brotherhood.  Now is the time 
to make justice a reality for all of God's children. 
 
It would be fatal for the nation to overlook the urgency of the moment. This sweltering  summer of the Negro's 
legitimate  discontent  will not pass until there is an invigorating  autumn of freedom and equality. Nineteen  sixty‐three is 
not an end, but a beginning.  And those who hope that the Negro needed to blow off steam and will now be content will 
have a rude awakening  if the nation returns to business as usual. And there will be neither rest nor tranquility  in America 
until the Negro is granted his citizenship  rights. The whirlwinds  of revolt will continue to shake the foundations  of our 
nation until the bright day of justice emerges.  
 But there is something  that I must say to my people, who stand on the warm threshold  which leads into the palace of 
justice: In the process of gaining our rightful place, we 
must not be guilty of wrongful deeds. Let us not seek to satisfy 
our thirst for freedom by drinking from the cup of bitterness  and hatred. We must forever conduct our struggle on the 
high plane of dignity and discipline.  We must not allow our creative protest to degenerate  into physical violence. Again 
and again, we must rise to the majestic heights of meeting physical force with soul force. 
 The marvelous  new militancy which has engulfed the Negro community  must not lead us to a distrust of all white 
people, for many of our white brothers,  as evidenced  by their presence 
here today, have come to realize that their 
destiny is tied up with our destiny. And they have come to realize that their freedom is inextricably  bound to our 
freedom.  
 We cannot walk alone. 
 "


CONCISE SUMMARY:
Prompt after formatting:
Write a concise summary of the following:


"And as we walk, we must make the pledge that we shall always march ahead. 
 We cannot turn back. 
 
There are those who are asking the devotees of civil rights, "When will you be satisfied?"  We can never be satisfied as 
long as the Negro is the victim of the 
unspeakable  horrors of police brutality. We can never be satisfied as long as our 
bodies, heavy with the fatigue of travel, cannot gain lodging in the motels of the highways  and the hotels of the cities. 
We cannot be satisfied as long as the negro's basic mobility is from a smaller ghetto to a larger one. We can never be 
satisfied as long as our children are stripped of their self‐hood and robbed of their dignity by signs stating: "For Whites 
Only." We cannot be satisfied as long as a Negro in Mississippi  cannot vote and a Negro in New York believes he has 
nothing for which to vote. No, no, we are not satisfied, and we will not be satisfied until "justice rolls down like waters, 
and righteousness  like a mighty stream."¹  
 
I am not unmindful  that some of you have come here out of great trials and tribulations.  Some of you have come fresh 
from narrow jail cells. And some of you have come from areas where your quest ‐‐ quest for freedom left you battered 
by the storms of persecution  and staggered  by the winds of police brutality. You have been the veterans of creative 
suffering.  Continue to work with the faith that unearned  suffering is redemptive.  Go back to Mississippi,  go back to 
Alabama,  go back to South Carolina, go back to Georgia, go back to Louisiana,  go back to the slums and ghettos of our 
northern cities, knowing that somehow  this situation can and will be changed. 
 
Let us not wallow in the valley of despair, I say to you today, my friends. 
 And so even though we face the difficulties  of today and tomorrow,  I still have a dream. It is a dream deeply rooted in 
the American  dream. 
 I have a dream that one day 
this nation will rise up and live out the true meaning of its creed: "We hold these truths to 
be self‐evident, that all men are created equal." 
 
I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners 
will be able to sit down together at the table of brotherhood.  
 I have a dream that one day even the state of Mississippi,  a state sweltering  with the heat of injustice, sweltering  with 
the heat of oppression,  will be transformed  into an oasis of freedom and justice.
 
 I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their 
skin but by the content of their character.  
 
I have a dream today! 
 I have a dream that one day, down in Alabama,
 with its vicious racists, with its governor having his lips dripping with the 
words of "interposition"  and "nullification"  ‐‐ one day right there in Alabama little black boys and black girls will be able 
to join hands with little white boys and white girls as sisters and brothers.  
 
I have a dream today! 
 I have a dream that one day every valley shall be exalted, and every hill and mountain  shall be made low, the rough 
places will be made plain, and the crooked places will be made straight; "and the glory of the Lord shall be revealed and 
all flesh 
shall see it together."2  
 This is our hope, and this is the faith that I go back to the South with. 
 With this faith, we will be able to hew out of the mountain  of despair a stone of hope. With this faith, we will be able to 
transform  the jangling
 discords of our nation into a beautiful symphony  of brotherhood.  With this faith, we will be able "


CONCISE SUMMARY:
Prompt after formatting:
Write a concise summary of the following:


"to work together,  to pray together,  to struggle together,  to go to jail together,  to stand up for freedom together,  
knowing that we will be free one day. 
 
And this will be the day ‐‐ this will be the day when all of God's children will be able to sing with new meaning:  
 My country 'tis of thee, sweet land of liberty, of thee I sing. 
 Land where my fathers died, land of the Pilgrim's pride, 
 
From every mountainside,  let freedom ring! 
 And if America is to be a great nation, this must become true. 
 
And so let freedom ring from
 the prodigious  hilltops of New Hampshire.  
 Let freedom ring from the mighty mountains  of New York. 
 Let freedom ring from the heightening  Alleghenies  of Pennsylvania.  
 
Let freedom ring from the snow‐capped Rockies of Colorado.  
 Let freedom ring from the curvaceous  slopes of California.  
 
But not only that: 
 Let 
freedom ring from Stone Mountain  of Georgia. 
 
Let freedom ring from Lookout Mountain  of Tennessee.  
 Let freedom ring from every hill and molehill of Mississippi.  
 From every mountainside,  let freedom ring. 
 
And when this happens, when we allow freedom ring, when we let it ring from every village and every
 hamlet, from 
every state and every city, we will be able to speed up that day when all of God's children, black men and white men, 
Jews and Gentiles, Protestants  and Catholics,  will be able to join hands and sing in the words of the old Negro spiritual: 
 
                 Free at last! Free at last! 
                  Thank God Almighty,  we are free at last!3 "


CONCISE SUMMARY:

> Finished chain.


> Entering new StuffDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
Write a concise summary of the following:


" Martin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people.

 Martin Luther King Jr. speaks of the need for civil rights and the struggle against police brutality, segregation, and racism. He encourages people to continue to fight for freedom and justice, and to not give up hope. He expresses his dream of a nation where everyone is judged by their character, not the color of their skin. He believes that with faith, hope, and perseverance, the nation can be transformed into a place of harmony and brotherhood.

 This passage is a call to action for people to come together and fight for freedom, knowing that one day they will be free. It encourages people to sing the words of the old Negro spiritual, "Free at last! Free at last! Thank God Almighty, we are free at last!" to celebrate their freedom."


CONCISE SUMMARY:

> Finished chain.

> Finished chain.

> Finished chain.

You can see the map reduce chain kicks off three summarization tasks for each page of the document, prompt starting with Write a concise summary of the following.... Once the subtasks are done, it actually enters a StuffDocumentChain which then puts the results of the subtasks together. Magical!

In [8]:

result["output_text"]

Out[8]:

' Martin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to advocate for civil rights and racial justice. He called for an end to segregation and discrimination, and urged people to fight peacefully and with dignity. He expressed his dream of a nation where everyone is judged by their character, not the color of their skin, and encouraged people to come together and fight for freedom.'

In [9]:

result["intermediate_steps"]

Out[9]:

[' Martin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people.',
 ' Martin Luther King Jr. speaks of the need for civil rights and the struggle against police brutality, segregation, and racism. He encourages people to continue to fight for freedom and justice, and to not give up hope. He expresses his dream of a nation where everyone is judged by their character, not the color of their skin. He believes that with faith, hope, and perseverance, the nation can be transformed into a place of harmony and brotherhood.',
 ' This passage is a call to action for people to come together and fight for freedom, knowing that one day they will be free. It encourages people to sing the words of the old Negro spiritual, "Free at last! Free at last! Thank God Almighty, we are free at last!" to celebrate their freedom.']

Refine Chain¶

Refine chain is slightly more fancy than map reduce. It produces the result for the first task, then provides the answer from the first task along with the second task to LLMs as context to generate the second result. This process repeats until it reaches the last task.

Refine uses a feedback loop to incrementally improve the quality of result each time. You can expect the answer to be more specific, detailed and relevant to the question. It comes with its own price as well, which is it could be quite time-consuming.

Let's take a look at how it works.

In [10]:

refine_chain = load_summarize_chain(llm=llm, chain_type="refine", verbose=True, return_intermediate_steps=True)
result = refine_chain(documents)

> Entering new RefineDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
Write a concise summary of the following:


"I HAVE A DREAM  
Martin Luther King, Jr. 
 
I am happy to join with you today in what will go down in history as the greatest demonstration  for freedom in the 
history of our nation. 
 Five score years ago, a great American,  in whose symbolic shadow we stand today, signed the Emancipation  
Proclamation.  This 
momentous  decree came as a great beacon light of hope to millions of Negro slaves who had been 
seared in the flames of withering  injustice. It came as a joyous daybreak  to end the long night of their captivity. 
 But one hundred years later, the Negro still is not free. One
 hundred years later, the life of the Negro is still sadly 
crippled by the manacles  of segregation  and the chains of discrimination.  One hundred years later, the Negro lives on a 
lonely island of poverty in the midst of a vast ocean of material prosperity.  One hundred years later, the Negro is still 
languished  in the corners of American  society and finds himself an exile in his own land. And so we've come here today 
to dramatize  a shameful  condition.  
 In a sense we've come to our nation's capital to cash a check. When the architects  of our republic wrote the
 magnificent  
words of the Constitution  and the Declaration  of Independence,  they were signing a promissory  note to which every 
American  was to fall heir. This note was a promise that all men, yes, black men as well as white men, would be 
guaranteed  the "unalienable  Rights" of "Life, Liberty and the pursuit of Happiness."  It is obvious today that America has 
defaulted  on this promissory  note, insofar as her citizens of color are concerned.  Instead of honoring this sacred 
obligation,  America has given the Negro people a bad check, a check which has come back marked "insufficient  funds." 
 
But we refuse to believe that the bank of justice is bankrupt.  We refuse to believe that there are insufficient  funds in the 
great vaults of opportunity  of this nation. And so, we've come to cash this check, a check that will give us upon demand 
the riches of freedom and the security of justice. 
 
We have also come to this hallowed spot to remind America of the fierce urgency of Now. This is no time to engage in 
the luxury of cooling off or to take the tranquilizing  drug of gradualism.  Now is the time to make real the promises of 
democracy.  Now is the time to rise from the dark and desolate valley of segregation  to the sunlit path of racial justice. 
Now is the time to lift our nation from the quicksands  of racial injustice to the solid rock of brotherhood.  Now is the time 
to make justice a reality for all of God's children. 
 
It would be fatal for the nation to overlook the urgency of the moment. This sweltering  summer of the Negro's 
legitimate  discontent  will not pass until there is an invigorating  autumn of freedom and equality. Nineteen  sixty‐three is 
not an end, but a beginning.  And those who hope that the Negro needed to blow off steam and will now be content will 
have a rude awakening  if the nation returns to business as usual. And there will be neither rest nor tranquility  in America 
until the Negro is granted his citizenship  rights. The whirlwinds  of revolt will continue to shake the foundations  of our 
nation until the bright day of justice emerges.  
 But there is something  that I must say to my people, who stand on the warm threshold  which leads into the palace of 
justice: In the process of gaining our rightful place, we 
must not be guilty of wrongful deeds. Let us not seek to satisfy 
our thirst for freedom by drinking from the cup of bitterness  and hatred. We must forever conduct our struggle on the 
high plane of dignity and discipline.  We must not allow our creative protest to degenerate  into physical violence. Again 
and again, we must rise to the majestic heights of meeting physical force with soul force. 
 The marvelous  new militancy which has engulfed the Negro community  must not lead us to a distrust of all white 
people, for many of our white brothers,  as evidenced  by their presence 
here today, have come to realize that their 
destiny is tied up with our destiny. And they have come to realize that their freedom is inextricably  bound to our 
freedom.  
 We cannot walk alone. 
 "


CONCISE SUMMARY:

> Finished chain.


> Entering new LLMChain chain...
Prompt after formatting:
Your job is to produce a final summary
We have provided an existing summary up to a certain point:  Martin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people.
We have the opportunity to refine the existing summary(only if needed) with some more context below.
------------
And as we walk, we must make the pledge that we shall always march ahead. 
 We cannot turn back. 
 
There are those who are asking the devotees of civil rights, "When will you be satisfied?"  We can never be satisfied as 
long as the Negro is the victim of the 
unspeakable  horrors of police brutality. We can never be satisfied as long as our 
bodies, heavy with the fatigue of travel, cannot gain lodging in the motels of the highways  and the hotels of the cities. 
We cannot be satisfied as long as the negro's basic mobility is from a smaller ghetto to a larger one. We can never be 
satisfied as long as our children are stripped of their self‐hood and robbed of their dignity by signs stating: "For Whites 
Only." We cannot be satisfied as long as a Negro in Mississippi  cannot vote and a Negro in New York believes he has 
nothing for which to vote. No, no, we are not satisfied, and we will not be satisfied until "justice rolls down like waters, 
and righteousness  like a mighty stream."¹  
 
I am not unmindful  that some of you have come here out of great trials and tribulations.  Some of you have come fresh 
from narrow jail cells. And some of you have come from areas where your quest ‐‐ quest for freedom left you battered 
by the storms of persecution  and staggered  by the winds of police brutality. You have been the veterans of creative 
suffering.  Continue to work with the faith that unearned  suffering is redemptive.  Go back to Mississippi,  go back to 
Alabama,  go back to South Carolina, go back to Georgia, go back to Louisiana,  go back to the slums and ghettos of our 
northern cities, knowing that somehow  this situation can and will be changed. 
 
Let us not wallow in the valley of despair, I say to you today, my friends. 
 And so even though we face the difficulties  of today and tomorrow,  I still have a dream. It is a dream deeply rooted in 
the American  dream. 
 I have a dream that one day 
this nation will rise up and live out the true meaning of its creed: "We hold these truths to 
be self‐evident, that all men are created equal." 
 
I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners 
will be able to sit down together at the table of brotherhood.  
 I have a dream that one day even the state of Mississippi,  a state sweltering  with the heat of injustice, sweltering  with 
the heat of oppression,  will be transformed  into an oasis of freedom and justice.
 
 I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their 
skin but by the content of their character.  
 
I have a dream today! 
 I have a dream that one day, down in Alabama,
 with its vicious racists, with its governor having his lips dripping with the 
words of "interposition"  and "nullification"  ‐‐ one day right there in Alabama little black boys and black girls will be able 
to join hands with little white boys and white girls as sisters and brothers.  
 
I have a dream today! 
 I have a dream that one day every valley shall be exalted, and every hill and mountain  shall be made low, the rough 
places will be made plain, and the crooked places will be made straight; "and the glory of the Lord shall be revealed and 
all flesh 
shall see it together."2  
 This is our hope, and this is the faith that I go back to the South with. 
 With this faith, we will be able to hew out of the mountain  of despair a stone of hope. With this faith, we will be able to 
transform  the jangling
 discords of our nation into a beautiful symphony  of brotherhood.  With this faith, we will be able 
------------
Given the new context, refine the original summary
If the context isn't useful, return the original summary.

> Finished chain.


> Entering new LLMChain chain...
Prompt after formatting:
Your job is to produce a final summary
We have provided an existing summary up to a certain point: 

Martin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people. He encouraged people to continue to work for civil rights, and to not give up hope for a better future. He envisioned a nation where people would be judged by the content of their character, not the color of their skin. He expressed his faith that with this dream, the nation could transform the jangling discords of our nation into a beautiful symphony of brotherhood.
We have the opportunity to refine the existing summary(only if needed) with some more context below.
------------
to work together,  to pray together,  to struggle together,  to go to jail together,  to stand up for freedom together,  
knowing that we will be free one day. 
 
And this will be the day ‐‐ this will be the day when all of God's children will be able to sing with new meaning:  
 My country 'tis of thee, sweet land of liberty, of thee I sing. 
 Land where my fathers died, land of the Pilgrim's pride, 
 
From every mountainside,  let freedom ring! 
 And if America is to be a great nation, this must become true. 
 
And so let freedom ring from
 the prodigious  hilltops of New Hampshire.  
 Let freedom ring from the mighty mountains  of New York. 
 Let freedom ring from the heightening  Alleghenies  of Pennsylvania.  
 
Let freedom ring from the snow‐capped Rockies of Colorado.  
 Let freedom ring from the curvaceous  slopes of California.  
 
But not only that: 
 Let 
freedom ring from Stone Mountain  of Georgia. 
 
Let freedom ring from Lookout Mountain  of Tennessee.  
 Let freedom ring from every hill and molehill of Mississippi.  
 From every mountainside,  let freedom ring. 
 
And when this happens, when we allow freedom ring, when we let it ring from every village and every
 hamlet, from 
every state and every city, we will be able to speed up that day when all of God's children, black men and white men, 
Jews and Gentiles, Protestants  and Catholics,  will be able to join hands and sing in the words of the old Negro spiritual: 
 
                 Free at last! Free at last! 
                  Thank God Almighty,  we are free at last!3 
------------
Given the new context, refine the original summary
If the context isn't useful, return the original summary.

> Finished chain.

> Finished chain.

Notice the first prompt is to Write a concise summary of the following:.... And the second and third prompts are Your job is to produce a final summary. We have provided an existing summary up to a certain point..., the summary up to a certain point refers to the result from previous executions.

And this is the final result it produces.

In [11]:

result["output_text"]

Out[11]:

'\n\nMartin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people. He encouraged people to continue to work for civil rights, and to not give up hope for a better future. He envisioned a nation where people would be judged by the content of their character, not the color of their skin. He expressed his faith that with this dream, the nation could transform the jangling discords of our nation into a beautiful symphony of brotherhood. He called for people to work, pray, struggle, and even go to jail together for the cause of freedom, and to never give up hope. He concluded his speech by expressing his faith that one day, all of God\'s children would be free and able to sing with new meaning the words of the old Negro spiritual, "Free at last! Free at last! Thank God Almighty'

In [12]:

result["intermediate_steps"]

Out[12]:

[' Martin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people.',
 '\n\nMartin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people. He encouraged people to continue to work for civil rights, and to not give up hope for a better future. He envisioned a nation where people would be judged by the content of their character, not the color of their skin. He expressed his faith that with this dream, the nation could transform the jangling discords of our nation into a beautiful symphony of brotherhood.',
 '\n\nMartin Luther King Jr. delivered his famous "I Have a Dream" speech in Washington D.C. to demonstrate for freedom and justice for African Americans. He argued that the promise of the Declaration of Independence and the Constitution had not been fulfilled for African Americans, and called for an end to segregation and discrimination. He urged people to rise up and fight for racial justice, and to do so peacefully and with dignity. He also reminded people that the freedom of African Americans is tied to the freedom of all people. He encouraged people to continue to work for civil rights, and to not give up hope for a better future. He envisioned a nation where people would be judged by the content of their character, not the color of their skin. He expressed his faith that with this dream, the nation could transform the jangling discords of our nation into a beautiful symphony of brotherhood. He called for people to work, pray, struggle, and even go to jail together for the cause of freedom, and to never give up hope. He concluded his speech by expressing his faith that one day, all of God\'s children would be free and able to sing with new meaning the words of the old Negro spiritual, "Free at last! Free at last! Thank God Almighty']

Re-Rank Chain¶

The last type of chain is Re-Rank. It's a confusing name at a glance, but it's actually quite straight forward. What re-rank chain does is using a ranking algorithm to reorder results generated based on their quality.

It would make much more sense when you think of its applications, such as search results, recommendations etc. The flip side of that is it's very sensitive to the quality of ranking algorithm. So use re-rank with caution.

Because there's nothing to be re-ranked in summarization task, I will use question-answer chain as example this time.

In [13]:

from langchain.chains.question_answering import load_qa_chain

rerank_chain = load_qa_chain(llm=llm, chain_type="map_rerank", verbose=True, return_intermediate_steps=True)
query = "What is the topic of Martin Luther King I Have a Dream Speech?"
result = rerank_chain({"input_documents":documents, "question":query}, return_only_outputs=True)

> Entering new MapRerankDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

In addition to giving an answer, also return a score of how fully it answered the user's question. This should be in the following format:

Question: [question here]
Helpful Answer: [answer here]
Score: [score between 0 and 100]

How to determine the score:
- Higher is a better answer
- Better responds fully to the asked question, with sufficient level of detail
- If you do not know the answer based on the context, that should be a score of 0
- Don't be overconfident!

Example #1

Context:
---------
Apples are red
---------
Question: what color are apples?
Helpful Answer: red
Score: 100

Example #2

Context:
---------
it was night and the witness forgot his glasses. he was not sure if it was a sports car or an suv
---------
Question: what type was the car?
Helpful Answer: a sports car or an suv
Score: 60

Example #3

Context:
---------
Pears are either red or orange
---------
Question: what color are apples?
Helpful Answer: This document does not answer the question
Score: 0

Begin!

Context:
---------
I HAVE A DREAM  
Martin Luther King, Jr. 
 
I am happy to join with you today in what will go down in history as the greatest demonstration  for freedom in the 
history of our nation. 
 Five score years ago, a great American,  in whose symbolic shadow we stand today, signed the Emancipation  
Proclamation.  This 
momentous  decree came as a great beacon light of hope to millions of Negro slaves who had been 
seared in the flames of withering  injustice. It came as a joyous daybreak  to end the long night of their captivity. 
 But one hundred years later, the Negro still is not free. One
 hundred years later, the life of the Negro is still sadly 
crippled by the manacles  of segregation  and the chains of discrimination.  One hundred years later, the Negro lives on a 
lonely island of poverty in the midst of a vast ocean of material prosperity.  One hundred years later, the Negro is still 
languished  in the corners of American  society and finds himself an exile in his own land. And so we've come here today 
to dramatize  a shameful  condition.  
 In a sense we've come to our nation's capital to cash a check. When the architects  of our republic wrote the
 magnificent  
words of the Constitution  and the Declaration  of Independence,  they were signing a promissory  note to which every 
American  was to fall heir. This note was a promise that all men, yes, black men as well as white men, would be 
guaranteed  the "unalienable  Rights" of "Life, Liberty and the pursuit of Happiness."  It is obvious today that America has 
defaulted  on this promissory  note, insofar as her citizens of color are concerned.  Instead of honoring this sacred 
obligation,  America has given the Negro people a bad check, a check which has come back marked "insufficient  funds." 
 
But we refuse to believe that the bank of justice is bankrupt.  We refuse to believe that there are insufficient  funds in the 
great vaults of opportunity  of this nation. And so, we've come to cash this check, a check that will give us upon demand 
the riches of freedom and the security of justice. 
 
We have also come to this hallowed spot to remind America of the fierce urgency of Now. This is no time to engage in 
the luxury of cooling off or to take the tranquilizing  drug of gradualism.  Now is the time to make real the promises of 
democracy.  Now is the time to rise from the dark and desolate valley of segregation  to the sunlit path of racial justice. 
Now is the time to lift our nation from the quicksands  of racial injustice to the solid rock of brotherhood.  Now is the time 
to make justice a reality for all of God's children. 
 
It would be fatal for the nation to overlook the urgency of the moment. This sweltering  summer of the Negro's 
legitimate  discontent  will not pass until there is an invigorating  autumn of freedom and equality. Nineteen  sixty‐three is 
not an end, but a beginning.  And those who hope that the Negro needed to blow off steam and will now be content will 
have a rude awakening  if the nation returns to business as usual. And there will be neither rest nor tranquility  in America 
until the Negro is granted his citizenship  rights. The whirlwinds  of revolt will continue to shake the foundations  of our 
nation until the bright day of justice emerges.  
 But there is something  that I must say to my people, who stand on the warm threshold  which leads into the palace of 
justice: In the process of gaining our rightful place, we 
must not be guilty of wrongful deeds. Let us not seek to satisfy 
our thirst for freedom by drinking from the cup of bitterness  and hatred. We must forever conduct our struggle on the 
high plane of dignity and discipline.  We must not allow our creative protest to degenerate  into physical violence. Again 
and again, we must rise to the majestic heights of meeting physical force with soul force. 
 The marvelous  new militancy which has engulfed the Negro community  must not lead us to a distrust of all white 
people, for many of our white brothers,  as evidenced  by their presence 
here today, have come to realize that their 
destiny is tied up with our destiny. And they have come to realize that their freedom is inextricably  bound to our 
freedom.  
 We cannot walk alone. 
 
---------
Question: What is the topic of Martin Luther King I Have a Dream Speech?
Helpful Answer:
Prompt after formatting:
Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

In addition to giving an answer, also return a score of how fully it answered the user's question. This should be in the following format:

Question: [question here]
Helpful Answer: [answer here]
Score: [score between 0 and 100]

How to determine the score:
- Higher is a better answer
- Better responds fully to the asked question, with sufficient level of detail
- If you do not know the answer based on the context, that should be a score of 0
- Don't be overconfident!

Example #1

Context:
---------
Apples are red
---------
Question: what color are apples?
Helpful Answer: red
Score: 100

Example #2

Context:
---------
it was night and the witness forgot his glasses. he was not sure if it was a sports car or an suv
---------
Question: what type was the car?
Helpful Answer: a sports car or an suv
Score: 60

Example #3

Context:
---------
Pears are either red or orange
---------
Question: what color are apples?
Helpful Answer: This document does not answer the question
Score: 0

Begin!

Context:
---------
And as we walk, we must make the pledge that we shall always march ahead. 
 We cannot turn back. 
 
There are those who are asking the devotees of civil rights, "When will you be satisfied?"  We can never be satisfied as 
long as the Negro is the victim of the 
unspeakable  horrors of police brutality. We can never be satisfied as long as our 
bodies, heavy with the fatigue of travel, cannot gain lodging in the motels of the highways  and the hotels of the cities. 
We cannot be satisfied as long as the negro's basic mobility is from a smaller ghetto to a larger one. We can never be 
satisfied as long as our children are stripped of their self‐hood and robbed of their dignity by signs stating: "For Whites 
Only." We cannot be satisfied as long as a Negro in Mississippi  cannot vote and a Negro in New York believes he has 
nothing for which to vote. No, no, we are not satisfied, and we will not be satisfied until "justice rolls down like waters, 
and righteousness  like a mighty stream."¹  
 
I am not unmindful  that some of you have come here out of great trials and tribulations.  Some of you have come fresh 
from narrow jail cells. And some of you have come from areas where your quest ‐‐ quest for freedom left you battered 
by the storms of persecution  and staggered  by the winds of police brutality. You have been the veterans of creative 
suffering.  Continue to work with the faith that unearned  suffering is redemptive.  Go back to Mississippi,  go back to 
Alabama,  go back to South Carolina, go back to Georgia, go back to Louisiana,  go back to the slums and ghettos of our 
northern cities, knowing that somehow  this situation can and will be changed. 
 
Let us not wallow in the valley of despair, I say to you today, my friends. 
 And so even though we face the difficulties  of today and tomorrow,  I still have a dream. It is a dream deeply rooted in 
the American  dream. 
 I have a dream that one day 
this nation will rise up and live out the true meaning of its creed: "We hold these truths to 
be self‐evident, that all men are created equal." 
 
I have a dream that one day on the red hills of Georgia, the sons of former slaves and the sons of former slave owners 
will be able to sit down together at the table of brotherhood.  
 I have a dream that one day even the state of Mississippi,  a state sweltering  with the heat of injustice, sweltering  with 
the heat of oppression,  will be transformed  into an oasis of freedom and justice.
 
 I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their 
skin but by the content of their character.  
 
I have a dream today! 
 I have a dream that one day, down in Alabama,
 with its vicious racists, with its governor having his lips dripping with the 
words of "interposition"  and "nullification"  ‐‐ one day right there in Alabama little black boys and black girls will be able 
to join hands with little white boys and white girls as sisters and brothers.  
 
I have a dream today! 
 I have a dream that one day every valley shall be exalted, and every hill and mountain  shall be made low, the rough 
places will be made plain, and the crooked places will be made straight; "and the glory of the Lord shall be revealed and 
all flesh 
shall see it together."2  
 This is our hope, and this is the faith that I go back to the South with. 
 With this faith, we will be able to hew out of the mountain  of despair a stone of hope. With this faith, we will be able to 
transform  the jangling
 discords of our nation into a beautiful symphony  of brotherhood.  With this faith, we will be able 
---------
Question: What is the topic of Martin Luther King I Have a Dream Speech?
Helpful Answer:
Prompt after formatting:
Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

In addition to giving an answer, also return a score of how fully it answered the user's question. This should be in the following format:

Question: [question here]
Helpful Answer: [answer here]
Score: [score between 0 and 100]

How to determine the score:
- Higher is a better answer
- Better responds fully to the asked question, with sufficient level of detail
- If you do not know the answer based on the context, that should be a score of 0
- Don't be overconfident!

Example #1

Context:
---------
Apples are red
---------
Question: what color are apples?
Helpful Answer: red
Score: 100

Example #2

Context:
---------
it was night and the witness forgot his glasses. he was not sure if it was a sports car or an suv
---------
Question: what type was the car?
Helpful Answer: a sports car or an suv
Score: 60

Example #3

Context:
---------
Pears are either red or orange
---------
Question: what color are apples?
Helpful Answer: This document does not answer the question
Score: 0

Begin!

Context:
---------
to work together,  to pray together,  to struggle together,  to go to jail together,  to stand up for freedom together,  
knowing that we will be free one day. 
 
And this will be the day ‐‐ this will be the day when all of God's children will be able to sing with new meaning:  
 My country 'tis of thee, sweet land of liberty, of thee I sing. 
 Land where my fathers died, land of the Pilgrim's pride, 
 
From every mountainside,  let freedom ring! 
 And if America is to be a great nation, this must become true. 
 
And so let freedom ring from
 the prodigious  hilltops of New Hampshire.  
 Let freedom ring from the mighty mountains  of New York. 
 Let freedom ring from the heightening  Alleghenies  of Pennsylvania.  
 
Let freedom ring from the snow‐capped Rockies of Colorado.  
 Let freedom ring from the curvaceous  slopes of California.  
 
But not only that: 
 Let 
freedom ring from Stone Mountain  of Georgia. 
 
Let freedom ring from Lookout Mountain  of Tennessee.  
 Let freedom ring from every hill and molehill of Mississippi.  
 From every mountainside,  let freedom ring. 
 
And when this happens, when we allow freedom ring, when we let it ring from every village and every
 hamlet, from 
every state and every city, we will be able to speed up that day when all of God's children, black men and white men, 
Jews and Gentiles, Protestants  and Catholics,  will be able to join hands and sing in the words of the old Negro spiritual: 
 
                 Free at last! Free at last! 
                  Thank God Almighty,  we are free at last!3 
---------
Question: What is the topic of Martin Luther King I Have a Dream Speech?
Helpful Answer:

> Finished chain.

> Finished chain.

In [14]:

result["output_text"]

Out[14]:

" The topic of Martin Luther King's I Have a Dream Speech is racial justice and equality."

In [15]:

result["intermediate_steps"]

Out[15]:

[{'answer': " The topic of Martin Luther King's I Have a Dream Speech is racial justice and equality.",
  'score': '100'},
 {'answer': " The topic of Martin Luther King's I Have a Dream Speech is civil rights and racial equality.",
  'score': '100'},
 {'answer': " The topic of Martin Luther King's I Have a Dream Speech is freedom and equality for all people.",
  'score': '100'}]

It shouldn't surprise you the chain actually does quite a decent job answering the question, despite the answer for each part of the document has a score of 100. It's a bit hard to compare with that score.

What's really interesting is re-rank actually implements a prompting technique called few-shot prompting. It would also be interesting to see how LangChain moves along with latest prompt research pagers, such as Chain-of-Thought and ToT, where I can see ToT is actively being worked on.

In [ ]:

prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

In addition to giving an answer, also return a score of how fully it answered the user's question. This should be in the following format:

Question: [question here]
Helpful Answer: [answer here]
Score: [score between 0 and 100]

How to determine the score:
- Higher is a better answer
- Better responds fully to the asked question, with sufficient level of detail
- If you do not know the answer based on the context, that should be a score of 0
- Don't be overconfident!

Example #1

Context:
---------
Apples are red
---------
Question: what color are apples?
Helpful Answer: red
Score: 100

Example #2

Context:
---------
it was night and the witness forgot his glasses. he was not sure if it was a sports car or an suv
---------
Question: what type was the car?
Helpful Answer: a sports car or an suv
Score: 60

Example #3

Context:
---------
Pears are either red or orange
---------
Question: what color are apples?
Helpful Answer: This document does not answer the question
Score: 0

Begin!

Context:
---------
{context}
---------
Question: {question}
Helpful Answer:"""