{"id":69577,"date":"2023-01-13T16:03:55","date_gmt":"2023-01-13T16:03:55","guid":{"rendered":"https:\/\/merikebi.warrenmyers.com\/?p=69577"},"modified":"2023-01-13T16:03:55","modified_gmt":"2023-01-13T16:03:55","slug":"answer-by-warren-for-splunk-use-streamstats-with-transaction-command-to-find-gaps-in-event-logs","status":"publish","type":"post","link":"https:\/\/merikebi.warrenmyers.com\/?p=69577","title":{"rendered":"Answer by warren for Splunk &#8211; Use streamstats with transaction command to find gaps in event logs"},"content":{"rendered":"<p>Without knowing what your data looks like, we can only guess, but something like the following should work:<\/p>\n<pre><code>index=ndx sourcetype=srctp message=* publicationID=*\n| fields - _raw\n| fields publicationID _time \n| sort 0 publicationID _time\n| eval message=if(match(message,&quot;Request.+publication&quot;),message+tostring(_time),message)\n| rex field=message &quot;Request.+publication(?&lt;begin&gt;.+)&quot;\n| filldown begin\n| stats min(_time) as start max(_time) as end by begin publicationID\n| eval gap=end-start\n| where gap&gt;3600\n| eval start=strftime(start,&quot;%c&quot;), end=strftime(end,&quot;%c&quot;)\n<\/code><\/pre>\n<p>What this does:<\/p>\n<ul>\n<li>a little housekeeping (keep only the fields we need, chuck the rest)<\/li>\n<li>[re]sort to ensure we&#8217;re grouped by <code>publicationID<\/code>, then by <code>_time<\/code> instead of <em>only<\/em> by <code>_time<\/code><\/li>\n<li>add a unique identified to <code>message<\/code> if it matches the pattern for being a beginning value<\/li>\n<li>extract a custom field (<code>begin<\/code>)<\/li>\n<li>if <code>message<\/code> doesn&#8217;t contain a value, <code>begin<\/code> will be null<\/li>\n<li><a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/SearchReference\/filldown\" rel=\"nofollow noreferrer\"><code>filldown<\/code><\/a> the value of <code>begin<\/code> (works just like Excel&#8217;s filldown functionality)<\/li>\n<li><code>stats<\/code>-out the start and end times of every individual <code>publicationID<\/code> event set (<em>a la<\/em> <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/SearchReference\/transaction\" rel=\"nofollow noreferrer\"><code>transaction<\/code><\/a>, but faster and more flexible)<\/li>\n<li>filter every line to ensure there is more than a 1 hour (3600 second) &quot;gap&quot; (<em>perhaps this would be better worded &quot;duration&quot;<\/em>?)<\/li>\n<li>format the start and end times for readability<\/li>\n<\/ul>\n<p>from User warren &#8211; Stack Overflow https:\/\/stackoverflow.com\/questions\/75110504\/splunk-use-streamstats-with-transaction-command-to-find-gaps-in-event-logs\/75111487#75111487<br \/>\nvia <a href=\"https:\/\/ifttt.com\/?ref=da&#038;site=wordpress\">IFTTT<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Without knowing what your data looks like, we can only guess, but something like the following should work: index=ndx sourcetype=srctp message=* publicationID=* | fields &#8211; _raw | fields publicationID _time | sort 0 publicationID _time | eval message=if(match(message,&quot;Request.+publication&quot;),message+tostring(_time),message) | rex field=message &quot;Request.+publication(?&lt;begin&gt;.+)&quot; | filldown begin | stats min(_time) as start max(_time) as end by begin &hellip;<br \/><a href=\"https:\/\/merikebi.warrenmyers.com\/?p=69577\" class=\"more-link pen_button pen_element_default pen_icon_arrow_double\">Continue reading <span class=\"screen-reader-text\">Answer by warren for Splunk &#8211; Use streamstats with transaction command to find gaps in event logs<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4],"tags":[991],"keyring_services":[],"class_list":["post-69577","post","type-post","status-publish","format-standard","hentry","category-blih","tag-stackexchange"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/69577","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=69577"}],"version-history":[{"count":1,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/69577\/revisions"}],"predecessor-version":[{"id":69578,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/69577\/revisions\/69578"}],"wp:attachment":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=69577"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=69577"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=69577"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fkeyring_services&post=69577"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}