全局搜索

In [1]

# Copyright (c) 2024 Microsoft Corporation.
# Licensed under the MIT License.

# 版权所有 (c) 2024 Microsoft Corporation。 # 根据 MIT 许可证获得许可。

In [2]

import os

import pandas as pd

from graphrag.config.enums import ModelType
from graphrag.config.models.language_model_config import LanguageModelConfig
from graphrag.language_model.manager import ModelManager
from graphrag.query.indexer_adapters import (
    read_indexer_communities,
    read_indexer_entities,
    read_indexer_reports,
)
from graphrag.query.structured_search.global_search.community_context import (
    GlobalCommunityContext,
)
from graphrag.query.structured_search.global_search.search import GlobalSearch
from graphrag.tokenizer.get_tokenizer import get_tokenizer

import os import pandas as pd from graphrag.config.enums import ModelType from graphrag.config.models.language_model_config import LanguageModelConfig from graphrag.language_model.manager import ModelManager from graphrag.query.indexer_adapters import ( read_indexer_communities, read_indexer_entities, read_indexer_reports, ) from graphrag.query.structured_search.global_search.community_context import ( GlobalCommunityContext, ) from graphrag.query.structured_search.global_search.search import GlobalSearch from graphrag.tokenizer.get_tokenizer import get_tokenizer

全局搜索示例¶

全局搜索方法通过以 Map-Reduce 方式搜索所有 AI 生成的社区报告来生成答案。这是一种资源密集型方法，但通常能为需要理解整个数据集的问题提供良好的响应（例如，本笔记本中提到的草药最重要的价值是什么？）。

LLM 设置¶

In [3]

api_key = os.environ["GRAPHRAG_API_KEY"]

config = LanguageModelConfig(
    api_key=api_key,
    type=ModelType.Chat,
    model_provider="openai",
    model="gpt-4.1",
    max_retries=20,
)
model = ModelManager().get_or_create_chat_model(
    name="global_search",
    model_type=ModelType.Chat,
    config=config,
)

tokenizer = get_tokenizer(config)

api_key = os.environ["GRAPHRAG_API_KEY"] config = LanguageModelConfig( api_key=api_key, type=ModelType.Chat, model_provider="openai", model="gpt-4.1", max_retries=20, ) model = ModelManager().get_or_create_chat_model( name="global_search", model_type=ModelType.Chat, config=config, ) tokenizer = get_tokenizer(config)

加载社区报告作为全局搜索的上下文¶

• 从 GraphRAG 的 community_reports 表中加载所有社区报告，用作全局搜索的上下文数据。
• 从 GraphRAG 的 entities 表中加载实体，用于计算社区权重以进行上下文排名。请注意，这是可选的（如果未提供实体，我们将不计算社区权重，仅使用社区报告表中的 rank 属性进行上下文排名）
• 从 GraphRAG 的 communities 表中加载所有社区，用于重建社区图层次结构以进行动态社区选择。

In [4]

# parquet files generated from indexing pipeline
INPUT_DIR = "./inputs/operation dulce"
COMMUNITY_TABLE = "communities"
COMMUNITY_REPORT_TABLE = "community_reports"
ENTITY_TABLE = "entities"

# community level in the Leiden community hierarchy from which we will load the community reports
# higher value means we use reports from more fine-grained communities (at the cost of higher computation cost)
COMMUNITY_LEVEL = 2

# 从索引管道生成的 parquet 文件 INPUT_DIR = "./inputs/operation dulce" COMMUNITY_TABLE = "communities" COMMUNITY_REPORT_TABLE = "community_reports" ENTITY_TABLE = "entities" # Leiden 社区层次结构中的社区级别，我们将从中加载社区报告 # 值越高表示我们使用来自更细粒度社区的报告（以更高的计算成本为代价） COMMUNITY_LEVEL = 2

In [5]

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet")
entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet")
report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet")

communities = read_indexer_communities(community_df, report_df)
reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL)
entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL)

print(f"Total report count: {len(report_df)}")
print(
    f"Report count after filtering by community level {COMMUNITY_LEVEL}: {len(reports)}"
)

report_df.head()

community_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_TABLE}.parquet") entity_df = pd.read_parquet(f"{INPUT_DIR}/{ENTITY_TABLE}.parquet") report_df = pd.read_parquet(f"{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet") communities = read_indexer_communities(community_df, report_df) reports = read_indexer_reports(report_df, community_df, COMMUNITY_LEVEL) entities = read_indexer_entities(entity_df, community_df, COMMUNITY_LEVEL) print(f"报告总数: {len(report_df)}") print( f"按社区级别 {COMMUNITY_LEVEL} 筛选后的报告数: {len(reports)}" ) report_df.head()

Total report count: 2
Report count after filtering by community level 2: 2

Out[5]

	id	human_readable_id	community	level	parent	children	title	summary	full_content	rank	rating_explanation	findings	full_content_json	period	size
0	6c3a555680d647ac8be866a129c7b0ea	0	0	0	-1	[]	操作：杜尔塞和杜尔塞基地探索	该社区围绕“操作：杜尔塞...”	# 操作：杜尔塞和杜尔塞基地探索\...	8.5	由于...，影响严重性评级为高	[{'explanation': '操作：杜尔塞是重要的...'}	{\n "title": "操作：杜尔塞和杜尔塞基...	2025-03-04	7
1	0127331a1ea34b8ba19de2c2a4cb3bc9	1	1	0	-1	[]	超自然军事小队和操作：杜尔塞	该社区以超自然军事小队为中心...	# 超自然军事小队和操作：杜尔塞...	8.5	由于...，影响严重性评级为高	[{'explanation': '特工 Alex Mercer 是关键角色...'}	{\n "title": "超自然军事小队和...	2025-03-04	9

基于社区报告构建全局上下文¶

In [6]

context_builder = GlobalCommunityContext(
    community_reports=reports,
    communities=communities,
    entities=entities,  # default to None if you don't want to use community weights for ranking
    tokenizer=tokenizer,
)

context_builder = GlobalCommunityContext( community_reports=reports, communities=communities, entities=entities, # 如果您不想使用社区权重进行排名，则默认为 None tokenizer=tokenizer, )

执行全局搜索¶

In [7]

context_builder_params = {
    "use_community_summary": False,  # False means using full community reports. True means using community short summaries.
    "shuffle_data": True,
    "include_community_rank": True,
    "min_community_rank": 0,
    "community_rank_name": "rank",
    "include_community_weight": True,
    "community_weight_name": "occurrence weight",
    "normalize_community_weight": True,
    "max_tokens": 12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    "context_name": "Reports",
}

map_llm_params = {
    "max_tokens": 1000,
    "temperature": 0.0,
    "response_format": {"type": "json_object"},
}

reduce_llm_params = {
    "max_tokens": 2000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 1000-1500)
    "temperature": 0.0,
}

context_builder_params = { "use_community_summary": False, # False 表示使用完整的社区报告。True 表示使用社区短摘要。 "shuffle_data": True, "include_community_rank": True, "min_community_rank": 0, "community_rank_name": "rank", "include_community_weight": True, "community_weight_name": "occurrence weight", "normalize_community_weight": True, "max_tokens": 12_000, # 根据您的模型令牌限制更改此值（如果您使用的是 8k 限制的模型，则一个好的设置可能是 5000） "context_name": "Reports", } map_llm_params = { "max_tokens": 1000, "temperature": 0.0, "response_format": {"type": "json_object"}, } reduce_llm_params = { "max_tokens": 2000, # 根据您的模型令牌限制更改此值（如果您使用的是 8k 限制的模型，则一个好的设置可能是 1000-1500） "temperature": 0.0, }

In [8]

search_engine = GlobalSearch(
    model=model,
    context_builder=context_builder,
    tokenizer=tokenizer,
    max_data_tokens=12_000,  # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
    map_llm_params=map_llm_params,
    reduce_llm_params=reduce_llm_params,
    allow_general_knowledge=False,  # set this to True will add instruction to encourage the LLM to incorporate general knowledge in the response, which may increase hallucinations, but could be useful in some use cases.
    json_mode=True,  # set this to False if your LLM model does not support JSON mode.
    context_builder_params=context_builder_params,
    concurrent_coroutines=32,
    response_type="multiple paragraphs",  # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report
)

search_engine = GlobalSearch( model=model, context_builder=context_builder, tokenizer=tokenizer, max_data_tokens=12_000, # 根据您的模型令牌限制更改此值（如果您使用的是 8k 限制的模型，则一个好的设置可能是 5000） map_llm_params=map_llm_params, reduce_llm_params=reduce_llm_params, allow_general_knowledge=False, # 将此设置为 True 将添加指令以鼓励 LLM 在响应中融入一般知识，这可能会增加幻觉，但在某些用例中可能很有用。 json_mode=True, # 如果您的 LLM 模型不支持 JSON 模式，请将此设置为 False。 context_builder_params=context_builder_params, concurrent_coroutines=32, response_type="multiple paragraphs", # 自由形式文本描述响应类型和格式，可以是任何内容，例如优先列表、单个段落、多个段落、多页报告 )

In [9]

result = await search_engine.search("What is operation dulce?")

print(result.response)

result = await search_engine.search("什么是杜尔塞行动？") print(result.response)

## Overview of Operation: Dulce

Operation: Dulce is a major mission undertaken by the Paranormal Military Squad, a specialized team tasked with investigating alien technology and its broader implications for humanity. The operation is centered on the exploration and investigation of the Dulce base, a highly secretive and mysterious location reputed to house advanced alien technology. The mission's complexity and significance make it a central focus for the community involved, as it connects all key entities and drives their actions [Data: Reports (0, 1)].

## Mission Objectives and Setting

The primary objective of Operation: Dulce is to navigate and uncover the secrets of the Dulce base. This facility is not only the main setting for the operation but also serves as the focal point for the team's efforts to understand and potentially secure alien technological assets. The exploration of the base is critical to achieving the operation's goals, as it may reveal information or artifacts with far-reaching consequences for humanity [Data: Reports (0, 1)].

## The Paranormal Military Squad

The operation is executed by the Paranormal Military Squad, an elite group composed of agents Alex Mercer, Taylor Cruz, Jordan Hayes, and Sam Rivera. Each member plays a crucial role in the mission, and their relationships and interactions with both the Dulce base and the alien technology are vital to the operation's dynamics and potential success. The team's expertise and cohesion are essential in navigating the challenges posed by the base and its secrets [Data: Reports (1)].

## Motivations and Implications

A strong sense of duty motivates the members of the Paranormal Military Squad to undertake Operation: Dulce. This sense of responsibility underscores the importance and complexity of the mission within the community. The operation is not only a technical or tactical endeavor but also a moral one, as the outcomes may have significant implications for the future of humanity and its relationship with alien technology [Data: Reports (0)].

## Conclusion

In summary, Operation: Dulce is a pivotal mission focused on the investigation of the Dulce base and its alien technology. It is carried out by the Paranormal Military Squad, whose members are driven by a profound sense of duty. The operation's success or failure may have lasting effects on humanity, making it a central and highly significant undertaking within the community [Data: Reports (0, 1)].

In [10]

# inspect the data used to build the context for the LLM responses
result.context_data["reports"]

# 检查用于为 LLM 响应构建上下文的数据 result.context_data["reports"]

Out[10]

	id	title	出现权重	内容	rank
0	1	超自然军事小队和操作：杜尔塞	1.0	# 超自然军事小队和操作：杜尔塞...	8.5
1	0	操作：杜尔塞和杜尔塞基地探索	1.0	# 操作：杜尔塞和杜尔塞基地探索\...	8.5

In [11]

# inspect number of LLM calls and tokens
print(
    f"LLM calls: {result.llm_calls}. Prompt tokens: {result.prompt_tokens}. Output tokens: {result.output_tokens}."
)

# 检查 LLM 调用次数和令牌数 print( f"LLM 调用次数: {result.llm_calls}。提示令牌数: {result.prompt_tokens}。输出令牌数: {result.output_tokens}。" )

LLM calls: 2. Prompt tokens: 3467. Output tokens: 779.