我是python的新手，我需要解析狀態代碼。我有一個解析HTTP日志文件的任務：按 IP 地址或 HTTP 狀態代碼（由用戶選擇）對記錄的請求進行分組。為每個組計算以下項之一（由用戶選擇）：請求計數所有已記錄請求的請求計數百分比傳輸的總字節數。我已經計算了請求和百分比。現在我不知道如何計算傳輸的字節（第3個任務）。日志文件的示例（此處在狀態代碼：6146、52315、12251、54662 之后顯示字節）：93.114.45.13 - - [17/May/2015:10:05:17 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"93.114.45.13 - - [17/May/2015:10:05:21 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"66.249.73.135 - - [17/May/2015:10:05:40 +0000] "GET /blog/tags/ipv6 HTTP/1.1" 200 12251 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)"83.149.9.216 - - [17/May/2015:10:05:25 +0000] "GET /presentations/logstash-monitorama-2013/images/elasticsearch.png HTTP/1.1" 200 8026 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"83.149.9.216 - - [17/May/2015:10:05:59 +0000] "GET /presentations/logstash-monitorama-2013/images/logstashbook.png HTTP/1.1" 200 54662 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
2 回答
慕婉清6462132
TA貢獻1804條經驗 獲得超2個贊
此解決方案不是使用多個正則表達式在日志中進行多次傳遞，而是使用一個正則表達式在一次傳遞中拉出所有相關值。
函數將日志的文本作為單個字符串傳遞。在下面的演示程序中，使用測試字符串。實際實現將調用此函數，并獲取讀取實際日志文件的結果。process_log
要跟蹤 IP 地址/狀態對，請使用 使用 a 作為default_factory。列表中的項目數計算 IP 地址/狀態組合的查看次數，每個列表項是為該 HTTP 請求傳輸的字節數。例如，字典的鍵/值對可能是：defaultdictlistip_status
key: ('123.12.11.9', '200') value: [6213, 9876, 376]對上述情況的解釋是，發現了3個IP地址“123.12.11.9”的狀態代碼“200”實例。為這 3 個實例傳輸的字節數為 6213、9876 和 376。
正則表達式的解釋：
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - .*?HTTP/1.1" (\d+) (\d+)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - -這本質上是OP用于識別IP地址的正則表達式，因此不需要太多的解釋。我遵循它以提供額外的后續上下文，以防萬一有其他類似外觀字符串的實例。IP 地址在捕獲組 1 中捕獲。- -.*?這將不貪婪地匹配 0 個或多個非換行符，直到以下內容。HTTP/1.1"匹配字符串以提供以下各項的左側上下文。HTTP/1.1"(\d+)匹配捕獲組 2 中的一個或多個數字（狀態）。匹配單個空格。
(\d+)匹配捕獲組 3 中的一個或多個數字（傳輸的字節）。
換句話說，我只是想確保我從正確的地方選擇正確的字段，方法是匹配我期望在我正在尋找的字段旁邊找到的內容。當您的正則表達式返回多個組時，通常比 . 返回一個迭代器，該迭代為每次迭代生成一個匹配對象。我添加了代碼來生成ip/狀態代碼/傳輸字節和狀態代碼的統計信息。您只需要一個或另一個，具體取決于用戶的需求。finditerfindallfinditer
代碼：
import re
from collections import defaultdict
def process_log(log):
ip_counter = defaultdict(list)
status_counter = defaultdict(int)
total_count = 0
for m in re.finditer(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - .*?HTTP/1.1" (\d+) (\d+)', log):
total_count += 1
ip = m[1]
status = m[2]
bytes = int(m[3])
ip_counter[(ip, status)].append(bytes)
status_counter[status] += 1
for k, v in ip_counter.items():
count = len(v)
percentage = count/total_count
total_bytes = sum(v)
ip = k[0]
status = k[1]
print(f"IP Address => {ip}, status => {status}, Count => {count}, Percentage => {percentage}, Total Bytes Transferred => {total_bytes}")
for k, v in status_counter.items():
count = v
percentage = count/total_count
print(f"Status Code => {k}, Percentage => {percentage}")
log = """93.114.45.13 - - [17/May/2015:10:05:17 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
93.114.45.13 - - [17/May/2015:10:05:21 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
66.249.73.135 - - [17/May/2015:10:05:40 +0000] "GET /blog/tags/ipv6 HTTP/1.1" 200 12251 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"83.149.9.216 - - [17/May/2015:10:05:25 +0000] "GET /presentations/logstash-monitorama-2013/images/elasticsearch.png HTTP/1.1" 200 8026 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [17/May/2015:10:05:59 +0000] "GET /presentations/logstash-monitorama-2013/images/logstashbook.png HTTP/1.1" 200 54662 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
"""
process_log(log)
指紋：
IP Address => 93.114.45.13, status => 200, Count => 2, Percentage => 0.4, Total Bytes Transferred => 58461
IP Address => 66.249.73.135, status => 200, Count => 1, Percentage => 0.2, Total Bytes Transferred => 12251
IP Address => 83.149.9.216, status => 200, Count => 2, Percentage => 0.4, Total Bytes Transferred => 62688
Status Code => 200, Percentage => 1.0
拉風的咖菲貓
TA貢獻1995條經驗 獲得超2個贊
要從日志文件中獲取傳輸的字節數，請執行以下操作：
def getBytes(filename):
with open(filename, 'r') as logfile:
for line in logfile:
regex = r'\/.+?\sHTTP\/1\..\"\s.{3}\s(.+?)\s'
bytesCount = re.search(regex, line)[1]
print("Bytes transfered: "+bytesCount)
添加回答
舉報
0/150
提交
取消