Basic BSim Queries
+ +In this section, we demonstrate some applications of our BSim database.
+ +Registering a BSim Database
+ +In order to query the database, you must register it with Ghidra:
+ +-
+
- From The Code Browser, Select BSim -> Manage Servers. +
- In the BSim Server Manager dialog, click the green plus
.
+ - Select the File radio button and use the chooser to select
example.mv.db
+ - Click OK +
- Click Dismiss to close the dialog. +
How to Query a BSim Database
+ +Before presenting the exercises, we describe the general mechanics of querying a BSim database.
+ +Initiating a BSim Query
+ +There are a number of ways to initiate a BSim query, including:
+ +-
+
- BSim -> Search Functions… from the Code Browser. +
- Right-click in the Listing and select BSim -> Search Functions… +
- Click on the BSim icon
in the Code Browser toolbar.
+
For these cases, the function(s) being queried depend on the current selection.
+If there is no selection, the function containing the current address is queried.
+If there is a selection, all functions whose entry points are within the selection are queried.
+An easy way to query all functions in a program is to select all addresses with Ctrl-A in the Listing window and then initiate a BSim query.
It is also possible to initiate a BSim query from the Decompiler window. +Simply right-click on a function name token and select BSim… to query the corresponding function. +This action is available on the name token in the decompiled function’s signature as well as tokens corresponding to names of callees.
+ +All of these actions bring up the BSim Search Dialog.
+ +The BSim Search Dialog
+ +From the BSim Search Dialog, you can
+ +-
+
- Select which BSim database to query. +
- Set query thresholds. +
- Bound the number of results returned for each function. +
- Set query filters. +
Selecting a BSim Database
+ +To query a registered BSim database, select that server from the BSim Server drop-down.
+ +Setting Query Options
+ +Similarity and confidence are scores used to evaluate the relationship between two vectors. +The respective fields in the dialog set lower bounds for these values for the matches returned by BSim.
+ +-
+
- Similarity
+
-
+
- Formally, the similarity of a match is the cosine of the angle between the vectors. +
- For BSim vectors, this value will always be between 0.0 and 1.0. +
- The higher the similarity score, the closer the vectors. +
+ - Confidence
+
-
+
- Intuitively, confidence quantifies the meatiness of a match. +
- Shared features increase this score and differing features decrease this score. +
- Sharing rare features contributes more to this score than sharing common features. +
- There is no upper bound for confidence when considered over all pairs of vectors. + However, if you fix a vector v, the greatest possible confidence score for a comparison involving v occurs when v is compared to itself. + The resulting confidence value is called the self-significance of v. +
+
Confidence is used to judge the significance of a match. +For example, many executables contain a function which simply returns a constant value. +Given two executables, each with such a function, the similarity score between the corresponding BSim vectors will be 1.0. +However, the confidence score of the match will be quite low, indicating that it is not very significant that the two executables “share” this code.
+ +In general, setting the thresholds involves a tradeoff: lower values mean that the database is more likely to return legitimate matches with significant differences, but also more likely to return matches which simply happen to share some features by chance. +The results of a BSim query can be sorted by the similarity and/or confidence of each match, so a common practice is to set the thresholds relatively low and to examine the matches in descending sort order.
+ +The Matches per Function bound controls the number of results returned for a single function. +Note that in large collections, certain small or common functions might have substantial numbers of identical matches.
+ +Filters are discussed in BSim Filters.
+ +Performing the Query
+ +Click the Search button in the dialog to perform a query.
+ +After successfully issuing a query, you will also see a Search Function(s) action (without the ellipsis) in certain contexts. +This will perform a BSim query on the selected functions using the same parameters as the last query (skipping the BSim Search Dialog).
+ +Exercises
+ +The database example contains vectors from a Linux executable used by Ghidra’s GNU demangler.
+Ghidra ships with several other versions of this executable.
+We use these different versions to demonstrate some of the capabilities of BSim.
Note: Use the default query settings and autoanalysis options for the exercises unless otherwise specified.
+ +Exercise: Function Identification
+ +-
+
- Import and analyze the binary
<ghidra_install_dir>/GPL/DemanglerGnu/os/win_x86_64/demangler_gnu_v2_41.exe. +-
+
- This executable is based on the same source code as
demangler_gnu_v2_41but compiled with Visual Studio instead of GCC.
+
+ - This executable is based on the same source code as
- Examine this binary in Ghidra and verify that the original function names are not present.
+
-
+
- Note that the function names are present in
demangler_gnu_v2_41.
+
+ - Note that the function names are present in
- Using the default query options, query
examplefor matches to the function at140006760.
+ - You should see the following search results:
+
+ -
+
- In this case, there is exactly one match, the similarity is 1.0, and the matching function has a non-default name (it won’t always be this easy). +
- The results window has two tables: the function-level results (upper table) and the executable-level results (lower table). + The executable-level results are covered in From Matching Functions to Matching Executables. +
+ - Right-click on the row of the match and perform the Compare Functions action to bring up the side-by-side comparison.
+
-
+
- The Listing View tab shows the disassembly. +
- The Decompiler Diff View tab shows the decompiled code. +
- Differences in the code are automatically highlighted in blue. +
- Either view can be toggled between a horizontal split and a vertical split using the drop-down menu. +
+ - Examine the diff views to verify that the match is valid. +
- Using the Apply Name action in the BSim Search Results table, apply the name from the search result to the queried function. +
Note: We cover the Decompiler Diff View in greater detail and discuss the various “Apply” actions in Evaluating Matches and Applying Information.
+ +Exercise: Changes to the Source Code
+ +-
+
- Import and analyze the executable
<ghidra_install_dir>/GPL/DemanglerGnu/os/linux_x86_64/demangler_gnu_v2_24. +-
+
- This executable is based on an earlier version of the source code than the executable in
example.
+
+ - This executable is based on an earlier version of the source code than the executable in
- Navigate to the function
expandargvindemangler_gnu_v2_24and issue a BSim query.
+ - What differences do you see in the decompiled code of the single match?
+ +
In demangler_gnu_v2_41...
The main differences are that call to dupargv is now in an if clause (and decompiler creates a related local variable) and there are two additional calls to free.
+ - The relevant source files are included with the Ghidra distribution:
+
-
+
<ghidra_install_dir>/GPL/DemanglerGnu/src/demangler_gnu_v2_24/c/argv.c
+ <ghidra_install_dir>/GPL/DemanglerGnu/src/demangler_gnu_v2_41/c/argv.c
+
+ - Verify that the differences you found are present in the source. +
Exercise: Cross-architectural Matching
+ +-
+
- Import and analyze the executable
+
<ghidra_install_dir>/GPL/DemanglerGnu/os/mac_arm_64/demangler_gnu_v2_41. +-
+
- This executable is based on the same source code as the executable in
examplebut compiled for a different architecture.
+ - Note: this file has the same name as the one we used to populate the BSim database, so you will have to give the resulting Ghidra program a different name or import it into a different directory in your Ghidra project. +
+ - This executable is based on the same source code as the executable in
- Navigate to
_expandargvand issue a BSim query. +In the decompiler diff view of the single match, what differences do you see regardingmemmoveandmemcpy? ++In the arm64 version...
In the arm64_version, the compiler replaced these functions with __memmove_chk and __memcpy_chk. The __chk versions have an extra parameter related to preventing buffer overflows. Neither the names nor the bodies of callees are incorporated into BSim signatures, but the arguments of a call are, so this change partly explains why the BSim vectors are not identical.
+ - Examine the Listing View tab and verify that the architectures are indeed different. +
A Remark on Query Thresholds and Indices
+ +Q: If you set the similarity and confidence thresholds to 0.0, will a BSim query return all of the functions in the database?
+ +A: No, because
+-
+
- For indexed databases (i.e., PostgreSQL and Elasticsearch), the index is designed so that vector comparisons are only performed between vectors which are likely to be close. +Most vectors will not even be considered as potential matches for a given queried vector. +
- Regardless of database backed, matches are only shown if the confidence score is above the confidence threshold of the query. +The interface will not allow you to set a negative confidence threshold, but confidence scores can be negative. +
- The Matches per Function parameter also controls how many functions are returned. +
Next Section: Ghidra from the Command Line
+ diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Basic_Queries.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Basic_Queries.md index c14a6885c6..c56d29908a 100644 --- a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Basic_Queries.md +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Basic_Queries.md @@ -7,7 +7,7 @@ In this section, we demonstrate some applications of our BSim database. In order to query the database, you must register it with Ghidra: 1. From The Code Browser, Select **BSim -> Manage Servers**. -1. In the BSim Server Manager dialog, click the green plus. +1. In the BSim Server Manager dialog, click the green plus . 1. Select the **File** radio button and use the chooser to select ``example.mv.db`` 1. Click **OK** 1. Click **Dismiss** to close the dialog. @@ -27,7 +27,7 @@ There are a number of ways to initiate a BSim query, including: For these cases, the function(s) being queried depend on the current selection. If there is no selection, the function containing the current address is queried. If there is a selection, all functions whose entry points are within the selection are queried. -For example, to query all functions in the program, first select all addresses in the program via ``Ctrl-A`` in the Listing window. +An easy way to query all functions in a program is to select all addresses with ``Ctrl-A`` in the Listing window and then initiate a BSim query. It is also possible to initiate a BSim query from the Decompiler window. Simply right-click on a function name token and select **BSim...** to query the corresponding function. @@ -44,7 +44,7 @@ From the BSim Search Dialog, you can - Bound the number of results returned for each function. - Set query filters. - + #### Selecting a BSim Database @@ -86,7 +86,7 @@ Filters are discussed in [BSim Filters](BSimTutorial_Filters.md). Click the **Search** button in the dialog to perform a query. After successfully issuing a query, you will also see a **Search Function(s)** action (without the ellipsis) in certain contexts. -This will perform a BSim query on the selected functions using the same parameters as the last query (skipping the BSim Seach Dialog). +This will perform a BSim query on the selected functions using the same parameters as the last query (skipping the BSim Search Dialog). ## Exercises @@ -96,7 +96,7 @@ We use these different versions to demonstrate some of the capabilities of BSim. **Note**: Use the default query settings and autoanalysis options for the exercises unless otherwise specified. -### Exercise 1: Function Identification +### Exercise: Function Identification 1. Import and analyze the binary ``In demangler_gnu_v2_41...
The main differences are that call to dupargv is now in an if clause (and decompiler creates a related local variable) and there are two additional calls to free.In the arm64 version...
In the arm64_version, the compiler replaced these functions with __memmove_chk and __memcpy_chk. The __chk versions have an extra parameter related to preventing buffer overflows. Neither the names nor the bodies of callees are incorporated into BSim signatures, but the arguments of a call are, so this change partly explains why the BSim vectors are not identical.Creating and Populating a BSim Database from the Ghidra GUI
+ +This section explains how to create and populate an H2-backed BSim database from the Ghidra GUI.
+ +Creating the Database
+ +To create a BSim database, first create a directory on your file system to contain the database.
+ +Next, perform the following steps from the Ghidra Code Browser:
+ +-
+
- Run the Ghidra script
CreateH2BSimDatabaseScript.java.
+ - In the resulting dialog:
+
-
+
- Enter “example” in the Database Name field. +
- Select the new directory in the Database Directory field. +
- Don’t change any of the other fields. +
+ - Click OK. +
Populating the Database
+ +We now populate the database with an executable which is contained in the Ghidra distribution.
+ +-
+
- Import and analyze the executable
<ghidra_install_dir>/GPL/DemanglerGnu/os/linux_x86_64/demangler_gnu_v2_41using the default analysis options.
+ - Run the Ghidra script
AddProgramToH2BSimDatabaseScript.javaon this program. +-
+
- The script will ask you to select an H2 database file. Use
example.mv.dbin the database directory.
+
+ - The script will ask you to select an H2 database file. Use
- In general you can run this script on other programs to add their signatures to this database, but that’s not necessary for the exercises in the next section. +
Next Section: Basic BSim Queries
+ diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.html b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.html new file mode 100755 index 0000000000..2504edf8aa --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.html @@ -0,0 +1,22 @@ +Starting Ghidra and Enabling the BSim Plugin:
+ +To begin the tutorial, perform the following steps:
+ +-
+
- Launch Ghidra. +
- Create a new non-shared project for this tutorial. +
- Launch the Code Browser. +
To enable BSim, perform the following steps:
+ +-
+
- File -> Configure from the Code Browser. +
- Click on the
Configurelink of theBSimentry.
+ - In the resulting dialog, ensure that the checkbox for
BSimSearchPluginis checked.
+
Next Section: Creating and Populating a BSim Database from the GUI
+ diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.md index 429a6bb36a..8737cb6561 100644 --- a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.md +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.md @@ -12,7 +12,7 @@ To enable BSim, perform the following steps: 1. Click on the ``Configure`` link of the ``BSim`` entry. 1. In the resulting dialog, ensure that the checkbox for ``BSimSearchPlugin`` is checked. - + Next Section: [Creating and Populating a BSim Database from the GUI](BSimTutorial_Creating_Database_From_GUI.md) diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Evaluating_Matches.html b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Evaluating_Matches.html new file mode 100755 index 0000000000..f70196747b --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Evaluating_Matches.html @@ -0,0 +1,135 @@ +Evaluating Matches and Applying Information
+ +Summarizing what we’ve created over the last few sections, we now have:
+-
+
- A stripped executable (
postgres).
+ - A Ghidra project containing some object files with debug information1 used to build that executable. +
- A BSim database containing the BSim signatures of the object files. +
We now demonstrate using BSim to help reverse engineer postgres.
+While doing this, we’ll showcase some of the features available in the decompiler diff view.
Exercise: Exploring the Highlights
+ +Import and analyze the stripped postgres executable into the tutorial project, then perform the following steps:
-
+
- Select all functions in
postgresviaCtrl-Ain the Listing.
+ - Perform a BSim query of the database
example. +-
+
- Note: We use the results of this query in the following few exercises. + If you don’t close the BSim search results window, you won’t have to issue the query again. +
+ - Sort the rows by confidence and find the row with
grouping_planneras the matching function. +The corresponding function inpostgresshould have a default name.
+ - Examine this match in the side-by-side decompiler view. +Note that the matching function has better data type information due to the debug information. +
- Q: Why does the placement of the
doubleargument between the functions? ++Answer
Floating point values and integer/pointer values are passed in separate sets of registers. +Neither ordering is wrong since both are consistent with the instructions of the function. +The debug info records a specific signature (and ordering) for the function, which Ghidra applies. +In the version without debug information, the decompiler used heuristics to determine the function's signature.
+
For matches with a fair number of differences, the decompiler diff panel can get pretty colorful. +Furthermore, as you click around, tokens will gain and lose highlights of various colors. +It’s worth giving a brief explanation of when highlighting happens and what the different colors mean. +Some terminology: if you click on a token in a decompiler panel, that token becomes the focused token.
+ +
The colors:
+ +-
+
- Blue is used to highlight differences between the two functions. +
- Pink is used to highlight the focused token and its match. +
- Lavender is used to highlight the focused token when it does not have a match. +
- Orange is used to highlight the focused token when it is ineligible for match. +Certain tokens, such as whitespace tokens or tokens used in variable declarations, are never assigned matching tokens. +
Exercise: Locking and Unlocking Scrolling
+ +By default, scrolling in the diff window is synchronized. +This means that scrolling within one window will also scroll within the other window. +In the decompiler diff window, scrolling works by matching one line in the left function with one line in the right function. +The two functions are aligned using those lines. +Initially, the functions are aligned using the functions’ signatures.
+ +As you click around in either function, the “aligning lines” will change. +If the focused token has a match, the scrolling is re-centered based on the lines containing the matched tokens. +If the focused token does not have a match, the functions will be aligned using the closest token to the focused token which does have a match.
+ +Synchronized scrolling can be toggled using the 
and 
icons in the toolbar.
-
+
- Experiment with locking and unlocking synchronized scrolling. +
Exercise: Applying Signatures
+ +If you are satisfied with a given match, you might want to apply information about the matching function to the queried function. +For example, you might want to apply the name or signature of the function. +There are some subtleties which determine how much information is safe to apply. +Hence there are three actions available under the Apply From Other menu when you right-click in the left panel:
+ +-
+
- Function Name will apply the right function’s name and namespace to the function on the left. +
- Function Signature will apply the name, namespace, and “skeleton” data types. + Structure and union data types are not transferred. + Instead, empty placeholder structures are created. +
- Function Signature and Data Types will apply the name and signature with full data types. +This may result in many data types being imported into the program (consider structures which refer to other structures). +
Warning: You should be absolutely certain that the datatypes are the exactly the same before applying signatures and data types. +If there have been any changes to a datatype’s definition, you could end up bringing incorrect datatypes into a program, even using BSim matches with 1.0 similarity. +Applying full data types is also problematic for cross-architecture matches.
+ +-
+
- Since we know it’s safe, apply the function signature and data types to the left function. +
There are similarly-named actions available on rows of the Function Matches table in the BSim Search Results window. +The Status column contains information about which rows have had their matches applied.
+ +Exercise: Comparing Callees
+ +The token matching algorithm matches a function call in one program to a function call in another by considering the data flow into and out of the CALL instruction, but it does not do anything with the bodies of the callees.
+However, given a matched pair of calls, you can bring up a new comparison window for the callees with the Compare Matching Callees action.
-
+
- Click in the left panel of the decompile diff window and press
Ctrl-F.
+ - Enter
FUN_and search for matched function calls where the callee in the left window has a default name and the callee in the right window has a non-default name.
+ - Right-click on one of the matched tokens and perform the Compare Matching Callees action. +
- In the comparison of the callees, apply the function signature and data types from the right function to the left function. +Verify that the update is reflected in the decompiler diff view of the callers. +
Exercise: Multiple Comparisons
+ +The function shown in a panel is controlled by a drop-down menu at the top of the panel. +This can be useful when you’d like to evaluate multiple matches to a single function.
+ +Exercise:
+ +-
+
- In the BSim Search Results window, right-click on a table column name, select Add/Remove Columns, and enable the Matches column. +
- Find two functions in
postgres, each of which has exactly two matches. +Select the corresponding four rows in the matches table and perform the Compare Functions action.
+ - Experiment with the drop-downs in each panel. +
In the next section, we discuss the Executable Results table.
+ +Next Section: From Matching Functions to Matching Executables
+-
+
-
+
Having debug information isn’t necessary to use BSim (as we’ve seen in a previous exercise), but it is convenient. Note that applying debug information can change BSim signatures, which can negatively impact matching between functions with debug information and functions without it. ↩
+
+
Answer
Floating point values and integer/pointer values are passed in separate sets registers. +Answer
Floating point values and integer/pointer values are passed in separate sets of registers. Neither ordering is wrong since both are consistent with the instructions of the function. The debug info records a specific signature (and ordering) for the function, which Ghidra applies. In the version without debug information, the decompiler used heuristics to determine the function's signature.From Matching Functions to Matching Executables
+ +In this section, we discuss the Executable Results table. +Each row of this table corresponds to one executable in the database. +The information in one row is an aggregation of all of the function-level matches into that row’s executable. +Your Executable Results table from the previous query should look similar to the following:
+ +
If you select a single row in the table and right-click on it, you will see the following actions:
+ +-
+
- Load Executable +Opens a read-only copy of the program in the Code Browser. +
- Filter on this Executable +Applies a filter which restricts the matches shown in the Function Matches table to matches which occur in the given executable. +
Exercise
+ +-
+
- Sort the Executable results by descending Function Count.
+An entry in this column shows the number of queried functions which have at least one match in the row’s executable (if
foohas 2 or more matches into a given executable, it still only contributes 1 to the function count). +What position isdemangler_gnu_v2_41? ++In this table...
It's in the first position.
+ - An entry in the Confidence column shows the sum of the confidence scores of all matches into the corresponding executable.
+If
foohas more than one match into a given executable, only the one with the highest (function-level) confidence contributes to the (executable-level) confidence score. +Sort the Executable results by descending confidence and observe thatdemangler_gnu_v2_41is now much further down the list. ++What could explain this?
If there are many function matches but the sum of all the confidences is relatively low, it is likely that many of the matches involve small functions with common BSim signatures.
+ - In the Executable match table, right click on
demangler_gnu_v2_41and apply the filter action. +Sort the filtered function matches by descending confidence. +Starting at the top, examine some of the matches and convince yourself that the given explanation is correct. +-
+
- Note: You can remove the filter using the Filter Results icon
in the toolbar.
+ We’ll discuss this further in BSim Filters
+
+ - Note: You can remove the filter using the Filter Results icon
From this exercise, we see that unrelated functions can be duplicates of each other, either because they are small or because they perform a common generic action. +Keep in mind that such functions can “pollute” the results of a blanket query. +In the next section, we demonstrate a technique to restrict queries to functions which are more likely to have meaningful matches.
+ +Next Section: Overview Queries
diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Exe_Results.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Exe_Results.md index c0768f40f9..bbe37b0184 100644 --- a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Exe_Results.md +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Exe_Results.md @@ -23,15 +23,15 @@ If you select a single row in the table and right-click on it, you will see the 1. An entry in the **Confidence** column shows the sum of the confidence scores of all matches into the corresponding executable. If ``foo`` has more than one match into a given executable, only the one with the highest (function-level) confidence contributes to the (executable-level) confidence score. Sort the Executable results by descending confidence and observe that ``demangler_gnu_v2_41`` is now much further down the list. -What could explain this?
If there are many function matches but the sum of all the confidences is relatively low, it is likely that many of the matches involve small functions. For such a match, it is more likely that the functions agree by chance rather than being derived from the same source code.What could explain this?
If there are many function matches but the sum of all the confidences is relatively low, it is likely that many of the matches involve small functions with common BSim signatures.BSim Filters
+ +There are a number of filters that can be applied to BSim queries, involving names, architectures, compilers, ingest dates, user-defined executable categories, and other attributes.
+ +Filters be can applied server-side or client-side.
+Server-side filters affect the query results sent to Ghidra from a BSim server and can be applied using the Filters drop-down in the BSim Search dialog.
+Client-side filters apply to the BSim Search results table and can be added and removed at will using the Filter Results icon .
+However, to “undo” a server-side filter, you have to issue another BSim query without the filter.
Exercise: Filters
+ +-
+
- Select all functions in
postgresand bring up the BSim Search dialog.
+ - Apply an Executable name does not equal filter with
demangler_gnu_v2_41as the name to exclude.
+ - Perform the query and verify
demangler_gnu_v2_41is not in the list of executables with matches.
+ - Using the Search Info icon
in the BSim Search Results toolbar, you can see the server-side filters applied to the query.
+Verify that this information is correct.
+ - Using the Filter Results icon , you can apply client-side filters to the query results. Experiment with applying and removing some client-side filters.
+
Next Section: Scripting and Visualization
diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Filters.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Filters.md index 8821bf5454..c6c9d7adfd 100644 --- a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Filters.md +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Filters.md @@ -1,13 +1,13 @@ # BSim Filters -There are a number of filters that can be applied to BSim queries, involving names, architectures, compilers, ingest dates, user-defined executable categories, and many other attributes. +There are a number of filters that can be applied to BSim queries, involving names, architectures, compilers, ingest dates, user-defined executable categories, and other attributes. Filters be can applied *server-side* or *client-side*. -Server-side filters affect the query results sent to Ghidra from a BSim server. -Client-side filters apply to the BSim Search results table and can be added and removed at will. -However, to "undo" a server-side filter, you have to issue an additional BSim query without the filter. +Server-side filters affect the query results sent to Ghidra from a BSim server and can be applied using the **Filters** drop-down in the BSim Search dialog. +Client-side filters apply to the BSim Search results table and can be added and removed at will using the **Filter Results** icon . +However, to "undo" a server-side filter, you have to issue another BSim query without the filter. + -Server-side filters can be applied using the **Filters** drop-down in the BSim Search dialog. ## Exercise: Filters diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.html b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.html new file mode 100755 index 0000000000..6630087b96 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.html @@ -0,0 +1,56 @@ +Ghidra Analysis from the Command Line
+ +For the remaining exercises, we need to populate our BSim database with a number of binaries. +We’d like a consistent set of binaries for the tutorial, but we don’t want to clutter the Ghidra distribution with dozens of additional executables. +Fortunately, the BSim plugin includes a script for building the PostgreSQL backend, and that build process creates hundreds of object files. +So we can just build PostgreSQL and harvest the object files we need.
+ +Note: For the tutorial, we continue to use the H2 BSim backend. +We do not run any PostgreSQL code, we simply analyze some files produced when building PostgreSQL.
+ +Note that these files must be built on a machine running Linux. +Windows users can build these files in a Linux virtual machine.
+ +First, download postgresql-15.3.tar.gz from the PostgreSQL web site.
+Put this file in <ghidra_install_dir>/Ghidra/Features/BSim.
To build the files, execute the following commands in a shell: 1
+ +cd <ghidra_install_dir>/Features/BSim
+export CFLAGS="-O2 -g"
+./make-postgres.sh
+mkdir ~/postgres_object_files
+cd build
+find . -name p*o -size +100000c -size -700000c -exec cp {} ~/postgres_object_files/ \;
+cd os/linux_x86_64/postgresql/bin
+strip -s postgres
+To continue on Windows, transfer the ~/postgres_object_files directory and the stripped postgres executable to your Windows machine.
Importing and Analyzing the Exercise Files
+ +Now that we have the executables, we can analyze them with the headless analyzer2. +The headless analyzer is distinct from BSim, but using it is the only feasible way to analyze substantial numbers of binaries.
+ +To analyze the files in Linux, execute the following commands in a shell.
+ +cd <ghidra_install_dir>/support
+./analyzeHeadless <ghidra_project_dir> postgres_object_files -import ~/postgres_object_files/*
+(On windows, use analyzeHeadless.bat and adjust paths accordingly.)
This will create a local Ghidra project called postgres_object_files in the directory <ghidra_project_dir>.
Next Section: BSim from the Command Line
+ +-
+
-
+
You may need to install additional packages and/or change some build options in order for PostgreSQL to build successfully. The error messages are generally informative. See the comments in
+make-postgres.sh. ↩
+ -
+
The headless analyzer has its own documentation:
+<ghidra_install_dir>/support/analyzeHeadlessREADME.html. ↩
+
Introduction to BSim
+ +As you’ve reverse engineered software, you’ve likely asked the following questions:
+ +-
+
- Which libraries were statically linked into this executable? +
- Does this executable share some code with another executable that I’ve analyzed? +
- What are the differences between version 1 and version 2 of a given executable? +
- Does this executable share code with another executable in a large collection of binaries? +
- Was this function pulled from an open-source library? +
BSim is intended to help with these questions (and others) by providing a way to search collections of binaries for similar, but not necessarily identical, functions.
+ +How Does BSim Work?
+ +The idea behind BSim is to generate a feature vector for each function in a binary. +The vectors are generated by Ghidra’s decompiler. +Each feature represents a small piece of data flow and/or control flow of the associated function. +The decompiler normalizes the feature vector representation so that different, but functionally equivalent, pieces of code often produce the same features. +Certain attributes, such as values of constants, names of registers, and data types, are intentionally not incorporated into the features.
+ +BSim vectors are compared using cosine similarity.
+Discrepancies between the vectors for foo and bar which are caused by differences in compilers, target architectures, and/or small changes to the source code typically result in vectors which are close but not identical.
BSim vectors can be stored in a dedicated database. +BSim databases intended to hold large1 numbers of vectors maintain an index based on locality-sensitive hashing. +The index drastically reduces the number of vector comparisons needed and allows for rapid retrieval of results.
+ +Querying foo against a BSim database typically yields a number of potential matches.
+Each individual match for foo can be compared to foo in a side-by-side view, and certain information (such as function name) can be quickly copied from a match to foo.
We frequently call BSim vectors the BSim signature of a function, or just the signature when the context is clear.
+ +Why “BSim”?
+ +We can think of each feature as representing a small piece of the behavior of a function, analogous to a snippet of source code. +Functions whose BSim vectors are close typically have many features in common, that is, they have similar behavior. +Hence the name “BSim”: Behavioral Similiarity.
+ +BSim Clients, BSim Databases, and Ghidra Projects
+ +Using BSim involves the following components:
+ +-
+
- A BSim Client, i.e., an instance of Ghidra with the BSim plugin enabled.
+
-
+
- This is where the reverse engineering happens. +
+ - A BSim Database, which stores the BSim signatures.
+
-
+
- Also stores some metadata about each function and its containing executable. +
- In particular, stores the ghidra:// URL of the associated Ghidra program. +
- Does not store disassembly or decompiled functions. +
+ - A Ghidra Project, which stores the analyzed programs used to populate the BSim database.
+
-
+
- Given a BSim match, the BSim client can use the ghidra:// URL to retrieve a program from a Ghidra project for side-by-side comparisons. +
- Note that a single BSim database can reference multiple Ghidra projects. +
+
Database Backends
+ +There are three supported database backends for BSim:
+ +-
+
-
+
PostgreSQL
+ +-
+
- The Ghidra distribution includes the source for a PostgreSQL plugin for BSim along with a + build script. +
- Users must download the PostgreSQL source. +
- Populated from shared Ghidra projects (i.e., requires a Ghidra server). +
- Server not supported on Windows (no restriction on clients). +
+ -
+
Elasticsearch
+ +-
+
- The
BSimElasticPluginextension contains an Elasticsearch plugin for BSim.
+ - This plugin must be installed into an existing Elasticsearch database. +
- Populated from shared Ghidra projects. +
+ - The
-
+
H2
+ +-
+
- Simplest way to use BSim:
+
-
+
- Backed by files on the user’s machine (don’t need to install database server). +
- Can be created and populated quickly. +
- Supported on all platforms. +
+ - Does not support large collections of binaries or multiple users. +
- Can be populated from non-shared (local) or shared Ghidra projects. +
+ - Simplest way to use BSim:
+
Next Section: Starting Ghidra and Enabling BSim
+ +-
+
-
+
Creating a database requires a database template, which determines the specifics of the index. Currently, Ghidra provides a medium template, intended for databases holding up to 10 million unique vectors, and a large template, intended for databases holding up to 100 million unique vectors. ↩
+
+
Overview Queries
+ +An Overview Query queries a BSim database for the number of matches to each function in an executable. +The matching functions themselves are not returned. +Similarity and Confidence thresholds can be set for an Overview Query, but there is no “Matches per Function” bound and no filters can be applied.
+ +To perform an Overview Query, select BSim -> Perform Overview… from the Code Browser.
+ +Exercise: Hit Counts and Self-Significance
+ +-
+
- Perform an Overview query on
postgresusing the default query thresholds. +You should see the following result: +
+ - Sort the table by the “Hit Count” column in ascending order. Typically, the functions with the largest hit counts will have low self-significance. +Verify that that is the case for this table. +
- Q: Examine the functions with the highest hit count. Why are there so many matches for these functions?
+ +
Answer:
These are all instances of PostgreSQL statistics-reporting functions. Their bodies are quite similar and they have identical BSim signatures.
+
Exercise: Selections and Queries
+ +Using the hit count column, it is possible to exclude functions with large numbers of matches.
+ +-
+
- In the Overview Table, select all functions whose hit count is 2 or less. +
- Right-click on the selection and perform the Search Selected Functions action.
+Sort the query results by descending Function Count and verify that
demangler_gnu_v2_41is far down the list.
+
Exercise: Vector Hashes
+ +Suppose foo and bar have the same number of hits in the Overview table.
+There are two possibilities:
-
+
fooandbarhave distinct feature vectors which happen to have the same number of matches.
+ fooandbarhave the same feature vector.
+
An optional column, Vector Hash, can be used to distinguish between these two cases.
+ +-
+
- Enable the Vector Hash Column in the Overview Table. +
- Find two functions with the same vector hash. +
- Select the two corresponding rows in the table and then transfer the selection to the Listing using the
icon in the BSim Overview toolbar.
+ - In the Listing, press
Shift-Cor right-click and perform the Compare Selected Functions action.
+ - In the resulting Function Comparison window, convince yourself that these two functions should have the same BSim signature. +
Next Section: Queries and Filters
diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Overview_Queries.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Overview_Queries.md index 2286673cee..b2034c01bc 100644 --- a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Overview_Queries.md +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Overview_Queries.md @@ -2,11 +2,11 @@ An **Overview Query** queries a BSim database for the number of matches to each function in an executable. The matching functions themselves are not returned. -Similarity and Confidence thresholds can be set for an Overview query, but there is no "Matches per Function" bound and no filters can be set. +Similarity and Confidence thresholds can be set for an Overview Query, but there is no "Matches per Function" bound and no filters can be applied. To perform an Overview Query, select **BSim -> Perform Overview...** from the Code Browser. -## Exercise 1: Hit Counts and Self-Significance. +## Exercise: Hit Counts and Self-Significance 1. Perform an Overview query on ``postgres`` using the default query thresholds. You should see the following result: @@ -14,9 +14,9 @@ You should see the following result: 1. Sort the table by the "Hit Count" column in ascending order. Typically, the functions with the largest hit counts will have low self-significance. Verify that that is the case for this table. 1. Q: Examine the functions with the highest hit count. Why are there so many matches for these functions? -Answer:
These are all instances of PostgreSQL statistics-reporting functions. Their bodies are quite similar.Answer:
These are all instances of PostgreSQL statistics-reporting functions. Their bodies are quite similar and they have identical BSim signatures.Scripting and Visualization
+ +Finally, we briefly mention a few other topics related to BSim.
+ +Scripting BSim
+ +There are are number of example scripts in the BSim script category, which demonstrate how to interact with BSim programmatically.
Visualizing Features
+ +Finally, if you’d like to see the particular BSim features in a function, you can use the BSim Feature Visualizer. +This plugin allows you to highlight regions of the decompiled code corresponding to a particular feature and to display a graph representing the feature.
+ +To use this plugin, first enable the BSimFeatureVisualizerPlugin via File -> Configure from the Code Browser.
+You can then bring it up via BSim -> BSim Feature Visualizer.
This is the end of the tutorial.
+ + diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Scripting.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Scripting.md index 37be02df9b..37f52720f7 100644 --- a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Scripting.md +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Scripting.md @@ -4,9 +4,9 @@ Finally, we briefly mention a few other topics related to BSim. ## Scripting BSim -There are are number of example scripts in the ``BSim`` script category, which demonstrate how to interact with BSim programmatically: +There are are number of example scripts in the ``BSim`` script category, which demonstrate how to interact with BSim programmatically. - + ## Visualizing Features @@ -14,10 +14,10 @@ Finally, if you'd like to see the particular BSim features in a function, you ca This plugin allows you to highlight regions of the decompiled code corresponding to a particular feature and to display a graph representing the feature. To use this plugin, first enable the ``BSimFeatureVisualizerPlugin`` via **File -> Configure** from the Code Browser. -You can then bring it via **BSim -> BSim Feature Visualizer**. +You can then bring it up via **BSim -> BSim Feature Visualizer**. - + This is the end of the tutorial. -[Return to the Beginning](README.md) \ No newline at end of file +[Return to the Beginning](README.md) diff --git a/GhidraDocs/GhidraClass/BSim/README.html b/GhidraDocs/GhidraClass/BSim/README.html new file mode 100755 index 0000000000..e6b4c0082f --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/README.html @@ -0,0 +1,24 @@ +BSim Tutorial
+ +BSim is a Ghidra plugin for finding structurally similar functions in (potentially large) collections of binaries. +It is based on Ghidra’s decompiler and can find matches across compilers, architectures, and/or small changes to source code.
+ +This tutorial demonstrates how create a small BSim database and walks through some typical use cases.
+ +Detailed information about BSim can be found in the “BSim” entry of the Ghidra Help.
+ +-
+
- Introduction to BSim +
- Starting Ghidra and Enabling BSim +
- Creating and Populating a BSim Database from the GUI +
- Basic BSim Queries +
- Ghidra from the Command Line +
- BSim from the Command Line +
- Evaluating Matches +
- From Matching Functions to Matching Executables +
- Overview Queries +
- BSim Filters +
- Scripting and Visualization +
Next Section: Introduction to BSim
diff --git a/GhidraDocs/GhidraClass/BSim/images/Plus2.png b/GhidraDocs/GhidraClass/BSim/images/Plus2.png new file mode 100644 index 0000000000..add4ad53dd Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/Plus2.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/script_manager.png b/GhidraDocs/GhidraClass/BSim/images/script_manager.png index 295704e53b..fac7a09989 100644 Binary files a/GhidraDocs/GhidraClass/BSim/images/script_manager.png and b/GhidraDocs/GhidraClass/BSim/images/script_manager.png differ diff --git a/GhidraDocs/certification.manifest b/GhidraDocs/certification.manifest index bc47c5fc0c..e6168f473a 100644 --- a/GhidraDocs/certification.manifest +++ b/GhidraDocs/certification.manifest @@ -21,18 +21,31 @@ GhidraClass/AdvancedDevelopment/GhidraAdvancedDevelopment.html||GHIDRA|||This fi GhidraClass/AdvancedDevelopment/GhidraAdvancedDevelopment_withNotes.html||Public Domain|||Slight modification of code that is available for distribution, without restrictions, (original extremely permissive wtf license allows us to change IP to Public Domain),from https://github.com/paulrouget/dzslides.|END| GhidraClass/AdvancedDevelopment/Images/GhidraLogo64.png||GHIDRA||||END| GhidraClass/AdvancedDevelopment/Images/highLevelClasses.png||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_BSim_Command_Line.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_BSim_Command_Line.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Basic_Queries.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Basic_Queries.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Creating_Database_From_GUI.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Creating_Database_From_GUI.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Enabling.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Enabling.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Evaluating_Matches.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Evaluating_Matches.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Exe_Results.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Exe_Results.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Filters.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Filters.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Intro.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Intro.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Overview_Queries.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Overview_Queries.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Scripting.html||GHIDRA||||END| GhidraClass/BSim/BSimTutorial_Scripting.md||GHIDRA||||END| +GhidraClass/BSim/README.html||GHIDRA||||END| GhidraClass/BSim/README.md||GHIDRA||||END| +GhidraClass/BSim/images/Plus2.png||GHIDRA||||END| GhidraClass/BSim/images/actions.png||GHIDRA||||END| GhidraClass/BSim/images/basic_query.png||GHIDRA||||END| GhidraClass/BSim/images/bsim_search_dialog.png||GHIDRA||||END|