Article
Towards a standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis
Contribution Summary
This paper addresses the need for a standardized approach to evaluate the performance of Large Language Models (LLMs) in digital forensic timeline analysis tasks. The proposed methodology includes a dataset, timeline generation, and ground truth development, and recommends the use of BLEU and ROUGE metrics for quantitative evaluation. The paper presents a case study using ChatGPT and demonstrates the effectiveness of the proposed methodology in evaluating LLM-based forensic timeline analysis. The study also discusses the limitations of applying LLMs to forensic timeline analysis and highlights the importance of maintaining the 'AI-assisted investigation' and 'human-in-the-loop' mantras when using LLMs in digital forensics.
Keywords: Large Language Models; Digital Forensic Timeline Analysis; Standardized Methodology; BLEU and ROUGE Metrics; ChatGPT; Forensic Science International: Digital Investigation; Digital Forensics; Artificial Intelligence
Abstract
Large language models (LLMs) have widespread adoption in many domains, including digital forensics. While prior research has largely centered on case studies and examples demonstrating how LLMs can assist forensic investigations, deeper explorations remain limited, i.e., a standardized approach for precise performance evaluations is lacking. Inspired by the NIST Computer Forensic Tool Testing Program, this paper proposes a standardized methodology to quantitatively evaluate the application of LLMs for digital forensic tasks, specifically in timeline analysis. The paper describes the components of the methodology, including the dataset, timeline generation, and ground truth development. In addition, the paper recommends the use of BLEU and ROUGE metrics for the quantitative evaluation of LLMs through case studies or tasks involving timeline analysis. Experimental results using ChatGPT demonstrate that the proposed methodology can effectively evaluate LLM-based forensic timeline analysis. Finally, we discuss the limitations of applying LLMs to forensic timeline analysis.
BibTeX
@article{Studiawan2025LLM-DF-Timeline-Analysis,
title = {Towards a standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis},
journal = {Forensic Science International: Digital Investigation},
volume = {54S},
pages = {301982},
month = 10,
year = {2025},
issn = {2666-2817},
doi = {https://doi.org/10.1016/j.fsidi.2025.301982},
author = {Studiawan, Hudan and Breitinger, Frank and Scanlon, Mark},
keywords = {LLM evaluation, Forensic timeline analysis, Large language models, ChatGPT, log2timeline/plaso},
abstract = {Large language models (LLMs) have widespread adoption in many domains, including digital forensics. While prior research has largely centered on case studies and examples demonstrating how LLMs can assist forensic investigations, deeper explorations remain limited, i.e., a standardized approach for precise performance evaluations is lacking. Inspired by the NIST Computer Forensic Tool Testing Program, this paper proposes a standardized methodology to quantitatively evaluate the application of LLMs for digital forensic tasks, specifically in timeline analysis. The paper describes the components of the methodology, including the dataset, timeline generation, and ground truth development. In addition, the paper recommends the use of BLEU and ROUGE metrics for the quantitative evaluation of LLMs through case studies or tasks involving timeline analysis. Experimental results using ChatGPT demonstrate that the proposed methodology can effectively evaluate LLM-based forensic timeline analysis. Finally, we discuss the limitations of applying LLMs to forensic timeline analysis.}
}