看起來您正在將所有文件的內容加載到內存中，然后再將它們寫回單個文件。這可以解釋為什么這個過程隨著時間的推移變得更慢。
優化該過程的一種方法是將讀取部分與寫入部分分開，并并行進行。這稱為生產者-消費者模式。Parallel它可以使用類、線程或任務來實現，但我將演示基于強大的TPL 數據流庫的實現，該庫特別適合此類作業。
private static async Task MergeFiles(IEnumerable<string> sourceFilePaths,
? ? string targetFilePath, CancellationToken cancellationToken = default,
? ? IProgress<int> progress = null)
{
? ? var readerBlock = new TransformBlock<string, string>(async filePath =>
? ? {
? ? ? ? return File.ReadAllText(filePath); // Read the small file
? ? }, new ExecutionDataflowBlockOptions()
? ? {
? ? ? ? MaxDegreeOfParallelism = 2, // Reading is parallelizable
? ? ? ? BoundedCapacity = 100, // No more than 100 file-paths buffered
? ? ? ? CancellationToken = cancellationToken, // Cancel at any time
? ? });
? ? StreamWriter streamWriter = null;
? ? int filesProcessed = 0;
? ? var writerBlock = new ActionBlock<string>(text =>
? ? {
? ? ? ? streamWriter.Write(text); // Append to the target file
? ? ? ? filesProcessed++;
? ? ? ? if (filesProcessed % 10 == 0) progress?.Report(filesProcessed);
? ? }, new ExecutionDataflowBlockOptions()
? ? {
? ? ? ? MaxDegreeOfParallelism = 1, // We can't parallelize the writer
? ? ? ? BoundedCapacity = 100, // No more than 100 file-contents buffered
? ? ? ? CancellationToken = cancellationToken, // Cancel at any time
? ? });
? ? readerBlock.LinkTo(writerBlock,
? ? ? ? new DataflowLinkOptions() { PropagateCompletion = true });
? ? // This is a tricky part. We use BoundedCapacity, so we must propagate manually
? ? // a possible failure of the writer to the reader, otherwise a deadlock may occur.
? ? PropagateFailure(writerBlock, readerBlock);
? ? // Open the output stream
? ? using (streamWriter = new StreamWriter(targetFilePath))
? ? {
? ? ? ? // Feed the reader with the file paths
? ? ? ? foreach (var filePath in sourceFilePaths)
? ? ? ? {
? ? ? ? ? ? var accepted = await readerBlock.SendAsync(filePath,
? ? ? ? ? ? ? ? cancellationToken); // Cancel at any time
? ? ? ? ? ? if (!accepted) break; // This will happen if the reader fails
? ? ? ? }
? ? ? ? readerBlock.Complete();
? ? ? ? await writerBlock.Completion;
? ? }
? ? async void PropagateFailure(IDataflowBlock block1, IDataflowBlock block2)
? ? {
? ? ? ? try { await block1.Completion.ConfigureAwait(false); }
? ? ? ? catch (Exception ex)
? ? ? ? {
? ? ? ? ? ? if (block1.Completion.IsCanceled) return; // On cancellation do nothing
? ? ? ? ? ? block2.Fault(ex);
? ? ? ? }
? ? }
}
使用示例：
var cts = new CancellationTokenSource();
var progress = new Progress<int>(value =>
{
? ? // Safe to update the UI
? ? Console.WriteLine($"Files processed: {value:#,0}");
});
var sourceFilePaths = Directory.EnumerateFiles(@"C:\SourceFolder", "*.log",
? ? SearchOption.AllDirectories); // Include subdirectories
await MergeFiles(sourceFilePaths, @"C:\AllLogs.log", cts.Token, progress);
BoundedCapacity用于控制內存使用。
如果磁盤驅動器是SSD，您可以嘗試使用MaxDegreeOfParallelism大于2的值讀取。
為了獲得最佳性能，您可以考慮寫入與包含源文件的驅動器不同的磁盤驅動器。
TPL 數據流庫可作為.NET Framework 的包提供，并且內置于 .NET Core。
