#diaphora — Public Fediverse posts

I have just published a new bug fixes minor release for #Diaphora, version 3.4.1.

https://github.com/joxeankoret/diaphora/releases/tag/3.4.1

I have published #Diaphora 3.4.0. Now you can install it in IDA by just running this:

$ hcli plugin install diaphora

https://github.com/joxeankoret/diaphora/releases/tag/3.4.0

Thank you very much to the person that donated 100 euros to the #Diaphora project, it's highly appreciated!

The Reverse Engineering community has spoken. #Diaphora will be ported to #Ghidra in the next months. I would love to have it working properly by the end of the year, but I cannot be sure. So, no ETA for now.

Thank you very much to the person that donated 25 euros to #Diaphora!

Question about IDA and Ghidra considering the latest IDA license changes, regarding #Diaphora: If a (highly integrated) Ghidra port of Diaphora appears "soon", would you...

Let me explain you the idea and the problem.

The Idea: Put a service for functions signatures using #Diaphora technologies online or a SQLite database file that can be easily downloaded somewhere for offline usage.

The Problem: Github doesn't allow files bigger than 100 MB. The initial signatures database I have is 1,2 GB, but it will grow a bit (I don't think it will be ever over 10 GB in years and years).

That explained, any recommendation about how to solve this problem with my resources?

What is a *cheap* way to put an online service that requires a database that will be under ~10 GB for a number of years? If it's free, better.

#Diaphora

I haven't published a new #Diaphora release just yet because the code is still a bit experimental and there are some problems that I'm trying to fix like, for example: SciKit Learn distributed models cannot be used if the version of the training machine and the machine where it's being used (where Diaphora runs) are different.

The code is also published (in github) already and #Diaphora now can use an already trained model to try to improve binary diffing results (matching). I haven't made yet a new release just yet as these changes are considered a bit experimental for now.

The datasets and tools for training and testing are here: https://github.com/joxeankoret/diaphora-ml
And Diaphora, is here: https://github.com/joxeankoret/diaphora

#Diaphora #BinaryDiffing #Bindiffing #ReverseEngineering #MachineLearning

Here are the slides of my "Simple Machine Learning Techniques for Binary Diffing (with Diaphora)" talk given at the @44CON conference last week:

https://github.com/joxeankoret/diaphora-ml/blob/main/docs/diaphora-ml-techniques-44con-final.pdf

#44con #Diaphora #MachineLearning #ReverseEngineering #BinaryDiffing

So, a classifier trained over the #Diaphora testing suite + the Cisco Talos Dataset-2 binaries works pretty well. More details on my talk next week at #44con

Thank you very much to the anonymous person who donated 100 euros to #Diaphora!

Why diff only assembly when you are using a decompiler for your work? #Diaphora

Grrrr... I still cannot find a model that fits all use-cases.

#Diaphora #MachineLearning

In case you are curious, I'm building a dataset out of cross comparison of the binaries in "Dataset-2" from this #Cisco #Talos dataset: https://github.com/Cisco-Talos/binary_function_similarity

The final idea is to train a model using the generated dataset (a simple CSV file) and then use it with #Diaphora.

PS: The cross comparison is only done *if* the binaries look to be the same.

Before optimising this thing, it used to foresee that for processing the same dataset it would take like ~2720 hours, and now around ~4 hours. This is acceptable. Finally. #Diaphora

>[Diaphora: Sun Sep 8 19:05:33 2024] Elapsed 0:02:59 second(s), remaining time ~4:08:39

I have just stumbled upon this post diffing some windows driver:

https://www.crowdfense.com/windows-wi-fi-driver-rce-vulnerability-cve-2024-30078/

Why use #BinDiff and see this [first picture] when you can use #Diaphora and see this [second picture]?

Of course, feel free to use whatever tool you prefer but, what's the point of doing more work? Diaphora finds out that only 2 functions are interesting for patch diffing and shows exactly, in the pseudo-code, what new chunk of code was added and what new function is being called. Diffing decompilation.

I had 3 potential ideas to apply machine learning techniques in #Diaphora:

1) Train a model with known good matches (best & partial) to try to improve results for less reliable heuristics.
2) Train a gigantic model with matches between binaries with and without symbols for GCC, Clang and MSVC, for Linux, Mac and Windows.
3) Train a model using versions of a same binary with symbols and without symbols, and then predict symbols in future versions without symbols.

Guess which one(s) got working?

Not bad results for a real use-case. #Diaphora #MachineLearning

With this data, I think I know which algorithm is the best fit for my use-case.

BTW, I'm surprised with the difference in times between the top 4 performing algorithms: most algorithms take more than 1 hour, but the decision trees classifier takes around 12 minutes.

#Diaphora #MachineLearning

First part done #Diaphora

I will explain in much more details what I'm working on, and I will also release the tools, during the upcoming @44CON conference in London.
#44con #Diaphora

So, how can I do this? Me, just a random open source developer, I have no option to do this at all. I don't have neither the required hardware resources at my "office" (at my home, my house), neither the money to "rent" the infrastructure from some big company, like Google Cloud, AWS, Azure, etc... Which means that, for non big companies, this project is not practical at all.

So, will I be able to finally provide trained models for #Diaphora? I really doubt so. But I will provide the tools.

Now, I have a dataset that I can use to train a model that I can use for #Diaphora. But this dataset is:

* A rather small subset of the Cisco Dataset.
* Only has Linux binaries for 3 architectures (x86, arm and mips, for 32 and 64) and 1 compiler (gcc).

In order to "properly" create a good dataset, it would need to...

* Include at least the 3 most used compilers: gcc, clang and msvc.
* Include binaries for the 3 most used operating systems.

Which means... a lot of binaries.

Let's suppose that the dataset I mention was just 1,000 binaries. Now, let's suppose that each binary has only 100 functions. To generate the good matches, I would need to take the binaries, using the symbols, generate some row with comparison data of the functions' features extracted by #Diaphora and add a new column with the value '1' to say "this row has the data for a good match".

Cool, this "just" means that I have to do 100 x 1,000 x 1,000 operations for that fictional dataset.

Now, it's as easy for me as telling #Diaphora to just analyse with IDA all binaries in the dataset and export the features Diaphora uses. After a couple of days, it finished and I had a bunch of Diaphora exported .sqlite files.

Cool. Next step? Generate a dataset of good and bad matches. And, oh boy... I think I'm going to just ignore the remaining space in this toot to go for the next one to explain...

OK, so now that the idea is clear, "rate functions matches between binaries to try to improve matching in #Diaphora", what can I do to solve this problem? The idea:

* Take a big binaries dataset.
* Export, with Diaphora, every single binary.
* Generate a dataset using as ground truth the symbols of the binaries.
* Train a model with the generated dataset.

Easy. Isn't it? How Hard Can It Be (TM)?

So, the idea is "try to use ML techniques to improve a real world tool, #Diaphora". But how? Basically, binary diffing (ignoring the part about extracting whatever features you want to use for comparison) can summarised in the following steps:

1) Find candidate matches.
2) Rate the matches into "good" or "bad".
3) Choose the best matches and generate a final output.

I haven't found any real world ML help for #1 or #3. But maybe it can help for #2?

Long thread ahead about training a classifier of "good/batch matches" for #Diaphora.

So, the whole idea that I have been working on for quite some time already to try to, somehow, improve matching in Diaphora is the following: Train a model to better determine if a pair of functions in two binaries (ie, a match between a function A in binary X, and function B in binary Y) is correct or not.