UIA控件检测

UIA控件检测是一种使用UI自动化(UIA)框架检测应用程序中标准控件的方法。它提供了一组API来访问和操作Windows应用程序中的UI元素。

注意

UIA控件检测可能无法检测应用程序中的非标准控件或自定义控件。

配置

要激活图标控件过滤,您需要在config_dev.yaml文件中将CONTROL_BACKEND设置为["uia"]

CONTROL_BACKEND: ["uia"]

参考

控件检查器的单例门面类。

初始化控件检查器。

参数
  • backend (str, 默认值: 'uia' ) –

    要使用的后端。

源代码在automator/ui_control/inspector.py
478
479
480
481
482
483
def __init__(self, backend: str = "uia") -> None:
    """
    Initialize the control inspector.
    :param backend: The backend to use.
    """
    self.backend = backend

desktop property

获取所有桌面窗口。

返回
  • UIAWrapper

    桌面的UIA包装器。

__new__(backend='uia')

单例模式。

源代码在automator/ui_control/inspector.py
467
468
469
470
471
472
473
474
475
476
def __new__(cls, backend: str = "uia") -> "ControlInspectorFacade":
    """
    Singleton pattern.
    """
    if backend not in cls._instances:
        instance = super().__new__(cls)
        instance.backend = backend
        instance.backend_strategy = BackendFactory.create_backend(backend)
        cls._instances[backend] = instance
    return cls._instances[backend]

find_control_elements_in_descendants(window, control_type_list=[], class_name_list=[], title_list=[], is_visible=True, is_enabled=True, depth=0)

在窗口的后代中查找控件元素。

参数
  • window (UIAWrapper) –

    要查找控件元素的窗口。

  • control_type_list (List[str], 默认值: [] ) –

    要查找的控件类型。

  • class_name_list (List[str], 默认值: [] ) –

    要查找的类名。

  • title_list (List[str], 默认值: [] ) –

    要查找的标题。

  • is_visible (bool, 默认值: True ) –

    控件元素是否可见。

  • is_enabled (bool, 默认值: True ) –

    控件元素是否启用。

  • depth (int, 默认值: 0 ) –

    要查找的后代深度。

返回
  • List[UIAWrapper]

    找到的控件元素。

源代码在automator/ui_control/inspector.py
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
def find_control_elements_in_descendants(
    self,
    window: UIAWrapper,
    control_type_list: List[str] = [],
    class_name_list: List[str] = [],
    title_list: List[str] = [],
    is_visible: bool = True,
    is_enabled: bool = True,
    depth: int = 0,
) -> List[UIAWrapper]:
    """
    Find control elements in descendants of the window.
    :param window: The window to find control elements.
    :param control_type_list: The control types to find.
    :param class_name_list: The class names to find.
    :param title_list: The titles to find.
    :param is_visible: Whether the control elements are visible.
    :param is_enabled: Whether the control elements are enabled.
    :param depth: The depth of the descendants to find.
    :return: The control elements found.
    """
    if self.backend == "uia":
        return self.backend_strategy.find_control_elements_in_descendants(
            window, control_type_list, [], title_list, is_visible, is_enabled, depth
        )
    elif self.backend == "win32":
        return self.backend_strategy.find_control_elements_in_descendants(
            window, [], class_name_list, title_list, is_visible, is_enabled, depth
        )
    else:
        return []

get_application_root_name(window) staticmethod

获取窗口的应用程序名称。

参数
  • window (UIAWrapper) –

    要获取应用程序名称的窗口。

返回
  • 字符串

    窗口的根应用程序名称。如果无法获取名称,则为空字符串("")。

源代码在automator/ui_control/inspector.py
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
@staticmethod
def get_application_root_name(window: UIAWrapper) -> str:
    """
    Get the application name of the window.
    :param window: The window to get the application name.
    :return: The root application name of the window. Empty string ("") if failed to get the name.
    """
    if window == None:
        return ""
    process_id = window.process_id()
    try:
        process = psutil.Process(process_id)
        return process.name()
    except psutil.NoSuchProcess:
        return ""

get_check_state(control_item) staticmethod

获取控件项的选中状态。param control_item: 要获取选中状态的控件项。return: 控件项的选中状态。

源代码在automator/ui_control/inspector.py
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
@staticmethod
def get_check_state(control_item: auto.Control) -> bool | None:
    """
    get the check state of the control item
    param control_item: the control item to get the check state
    return: the check state of the control item
    """
    is_checked = None
    is_selected = None
    try:
        assert isinstance(
            control_item, auto.Control
        ), f"{control_item =} is not a Control"
        is_checked = (
            control_item.GetLegacyIAccessiblePattern().State
            & auto.AccessibleState.Checked
            == auto.AccessibleState.Checked
        )
        if is_checked:
            return is_checked
        is_selected = (
            control_item.GetLegacyIAccessiblePattern().State
            & auto.AccessibleState.Selected
            == auto.AccessibleState.Selected
        )
        if is_selected:
            return is_selected
        return None
    except Exception as e:
        # print(f'item {control_item} not available for check state.')
        # print(e)
        return None

get_control_info(window, field_list=[]) staticmethod

获取窗口的控件信息。

参数
  • window (UIAWrapper) –

    要获取控件信息的窗口。

  • field_list (List[str], 默认值: [] ) –

    要获取的字段。return: 窗口的控件信息。

源代码在automator/ui_control/inspector.py
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
@staticmethod
def get_control_info(
    window: UIAWrapper, field_list: List[str] = []
) -> Dict[str, str]:
    """
    Get control info of the window.
    :param window: The window to get control info.
    :param field_list: The fields to get.
    return: The control info of the window.
    """
    control_info: Dict[str, str] = {}

    def assign(prop_name: str, prop_value_func: Callable[[], str]) -> None:
        if len(field_list) > 0 and prop_name not in field_list:
            return
        control_info[prop_name] = prop_value_func()

    try:
        assign("control_type", lambda: window.element_info.control_type)
        assign("control_id", lambda: window.element_info.control_id)
        assign("control_class", lambda: window.element_info.class_name)
        assign("control_name", lambda: window.element_info.name)
        rectangle = window.element_info.rectangle
        assign(
            "control_rect",
            lambda: (
                rectangle.left,
                rectangle.top,
                rectangle.right,
                rectangle.bottom,
            ),
        )
        assign("control_text", lambda: window.element_info.name)
        assign("control_title", lambda: window.window_text())
        assign("selected", lambda: ControlInspectorFacade.get_check_state(window))

        try:
            source = window.element_info.source
            assign("source", lambda: source)
        except:
            assign("source", lambda: "")

        return control_info
    except:
        return {}

get_control_info_batch(window_list, field_list=[])

获取窗口的控件信息。

参数
  • window_list (List[UIAWrapper]) –

    要获取控件信息的窗口列表。

  • field_list (List[str], 默认值: [] ) –

    要获取的字段。return: 窗口控件信息的列表。

源代码在automator/ui_control/inspector.py
566
567
568
569
570
571
572
573
574
575
576
577
578
def get_control_info_batch(
    self, window_list: List[UIAWrapper], field_list: List[str] = []
) -> List[Dict[str, str]]:
    """
    Get control info of the window.
    :param window_list: The list of windows to get control info.
    :param field_list: The fields to get.
    return: The list of control info of the window.
    """
    control_info_list = []
    for window in window_list:
        control_info_list.append(self.get_control_info(window, field_list))
    return control_info_list

get_control_info_list_of_dict(window_dict, field_list=[])

获取窗口的控件信息。

参数
  • window_dict (Dict[str, UIAWrapper]) –

    要获取控件信息的窗口字典。

  • field_list (List[str], 默认值: [] ) –

    要获取的字段。return: 窗口控件信息的列表。

源代码在automator/ui_control/inspector.py
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
def get_control_info_list_of_dict(
    self, window_dict: Dict[str, UIAWrapper], field_list: List[str] = []
) -> List[Dict[str, str]]:
    """
    Get control info of the window.
    :param window_dict: The dict of windows to get control info.
    :param field_list: The fields to get.
    return: The list of control info of the window.
    """
    control_info_list = []
    for key in window_dict.keys():
        window = window_dict[key]
        control_info = self.get_control_info(window, field_list)
        control_info["label"] = key
        control_info_list.append(control_info)
    return control_info_list

get_desktop_app_dict(remove_empty=True)

获取桌面上所有应用程序并以字典形式返回。

参数
  • remove_empty (bool, 默认值: True ) –

    是否删除空标题。

返回
  • Dict[str, UIAWrapper]

    桌面上应用程序的字典。

源代码在automator/ui_control/inspector.py
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
def get_desktop_app_dict(self, remove_empty: bool = True) -> Dict[str, UIAWrapper]:
    """
    Get all the apps on the desktop and return as a dict.
    :param remove_empty: Whether to remove empty titles.
    :return: The apps on the desktop as a dict.
    """
    desktop_windows = self.get_desktop_windows(remove_empty)

    desktop_windows_with_gui = []

    for window in desktop_windows:
        try:
            window.is_normal()
            desktop_windows_with_gui.append(window)
        except:
            pass

    desktop_windows_dict = dict(
        zip(
            [str(i + 1) for i in range(len(desktop_windows_with_gui))],
            desktop_windows_with_gui,
        )
    )
    return desktop_windows_dict

get_desktop_app_info(desktop_windows_dict, field_list=['control_text', 'control_type'])

获取桌面上所有应用程序的控件信息。

参数
  • desktop_windows_dict (Dict[str, UIAWrapper]) –

    桌面上应用程序的字典。

  • field_list (List[str], 默认值: ['control_text', 'control_type'] ) –

    要获取的应用程序信息字段。

返回
  • List[Dict[str, str]]

    桌面上所有应用程序的控件信息。

源代码在automator/ui_control/inspector.py
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
def get_desktop_app_info(
    self,
    desktop_windows_dict: Dict[str, UIAWrapper],
    field_list: List[str] = ["control_text", "control_type"],
) -> List[Dict[str, str]]:
    """
    Get control info of all the apps on the desktop.
    :param desktop_windows_dict: The dict of apps on the desktop.
    :param field_list: The fields of app info to get.
    :return: The control info of all the apps on the desktop.
    """
    desktop_windows_info = self.get_control_info_list_of_dict(
        desktop_windows_dict, field_list
    )
    return desktop_windows_info

get_desktop_windows(remove_empty=True)

获取桌面上所有应用程序。

参数
  • remove_empty (bool, 默认值: True ) –

    是否删除空标题。

返回
  • List[UIAWrapper]

    桌面上应用程序。

源代码在automator/ui_control/inspector.py
485
486
487
488
489
490
491
def get_desktop_windows(self, remove_empty: bool = True) -> List[UIAWrapper]:
    """
    Get all the apps on the desktop.
    :param remove_empty: Whether to remove empty titles.
    :return: The apps on the desktop.
    """
    return self.backend_strategy.get_desktop_windows(remove_empty)