{"id":18502,"date":"2020-08-18T19:35:19","date_gmt":"2020-08-18T19:35:19","guid":{"rendered":"https:\/\/merikebi.warrenmyers.com\/?p=18502"},"modified":"2020-08-18T19:35:19","modified_gmt":"2020-08-18T19:35:19","slug":"how-to-timechart-possibly-better-than-timechart-in-splunk","status":"publish","type":"post","link":"https:\/\/merikebi.warrenmyers.com\/?p=18502","title":{"rendered":"how-to timechart [possibly] better than timechart in splunk"},"content":{"rendered":"<p>I recently had cause to do an extensive <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/lastest\/Viz\/VisualizationTrellis\">trellised<\/a> <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/SearchReference\/timechart\">timechart<\/a> for a dashboard at $CUSTOMER in <a href=\"https:\/\/smile.amazon.com\/s\/?tag=antipaucity-20&#038;creative=392009&#038;campaign=212361&#038;field-keywords=Splunk\">Splunk<\/a>.<\/p>\n<div class=\"wp-block-image is-style-rounded\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" src=\"https:\/\/www.splunk.com\/content\/dam\/splunk2\/images\/logos\/splunk-logo.svg\" alt=\"\"\/><\/figure>\n<\/div>\n<p>They have a couple hundred locations reporting networked devices.<\/p>\n<p>I needed to report on how many devices they&#8217;ve reported every day over the last 90 days (I would have <em>liked<\/em> to go back further&#8230;but retention is only 90 days on this data).<\/p>\n<h4>My initial instinct was to do this:<\/h4>\n<div class=\"wp-block-group\">\n<div class=\"wp-block-group__inner-container\">\n<p><code>index=ndx sourcetype=srctp site=* ip=* earliest=-90d<br \/>| timechart limit=0 span=1d dc(ip) by site<\/code><\/p>\n<\/div>\n<\/div>\n<p>Except&#8230;that takes <strong>well over<\/strong> an hour to run &#8211; so the job gets terminated at ~60 minutes.<\/p>\n<p>What possible other approaches could be made? <\/p>\n<h1 class=\"has-text-align-center\"><img decoding=\"async\" src=\"https:\/\/s.w.org\/images\/core\/emoji\/13.0.0\/72x72\/1f914.png\" alt=\"?\" class=\"wp-smiley\" style=\"height: 1em; max-height: 1em;\" \/><\/h1>\n<p>Well. <\/p>\n<h4>Here are a few that I thought about:<\/h4>\n<ol>\n<li>Use <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/SearchReference\/multisearch\"><code>multisearch<\/code><\/a>, and group 9 10d searches together.\n<ul>\n<li>I&#8217;ve done things like this before with good success. But it&#8217;s &#8230; ugly. Very, very ugly.<\/li>\n<li>You can <em>almost<\/em> always accomplish what you want via <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/SearchReference\/stats\"><code>stats<\/code><\/a>, too &#8211; but it can be tricky.<\/li>\n<\/ul>\n<\/li>\n<li>Pre-populate a <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/Knowledge\/Aboutlookupsandfieldactions\">lookup<\/a> table with older data (<em>a la<\/em> option 1 above, but done &#8220;by hand&#8221;), and then just append &#8220;more recent&#8221; data onto the table in the future.\n<ul>\n<li>This would give the advantage of getting a longer history going forward<\/li>\n<li>Ensuring &#8220;cleanliness&#8221; of the table would require some <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/Capacity\/HowsavedsearchesaffectSplunkEnterpriseperformance\">maintenance<\/a> scheduled searches\/reports &#8230; but it&#8217;s doable<\/li>\n<\/ul>\n<\/li>\n<li>Something <em>else<\/em> &#8230; that &#8220;happens&#8221; to work like a <code>timechart<\/code> &#8211; but runs in an acceptable time frame.<\/li>\n<li>Try <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/SearchReference\/bin\"><code>bin<\/code><\/a>ning <code>_time<\/code>\n<ol>\n<li>Tried &#8211; didn&#8217;t work <img decoding=\"async\" src=\"https:\/\/s.w.org\/images\/core\/emoji\/13.0.0\/72x72\/1f928.png\" alt=\"?\" class=\"wp-smiley\" style=\"height: 1em; max-height: 1em;\" \/><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>So what <em><strong>did<\/strong><\/em> I do? <\/p>\n<h4>I asked for ideas.<\/h4>\n<p>If you&#8217;re regularly (or <strong><em>ir<\/em><\/strong>regularly) using <a href=\"https:\/\/smile.amazon.com\/s\/?tag=antipaucity-20&#038;creative=392009&#038;campaign=212361&#038;field-keywords=Splunk\">Splunk<\/a>, you should join the <a href=\"https:\/\/splunk-usergroups.slack.com\">Splunk Usergroups Slack<\/a>.<\/p>\n<div class=\"wp-block-image is-style-rounded\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" src=\"https:\/\/a.slack-edge.com\/236db6\/marketing\/img\/promos\/slack-101-learning-slack-made-simple@2x.jpg\" alt=\"\"\/><\/figure>\n<\/div>\n<p>Go join it now, if you&#8217;re not on it already.<\/p>\n<p>Don&#8217;t worry &#8211; this blog post will be here when you get back.<\/p>\n<p>You&#8217;ve joined? Good good. Look me up &#8211; I&#8217;m @Warren Myers. And I love to help when I can <img decoding=\"async\" src=\"https:\/\/s.w.org\/images\/core\/emoji\/13.0.0\/72x72\/1f920.png\" alt=\"?\" class=\"wp-smiley\" style=\"height: 1em; max-height: 1em;\" \/>.<\/p>\n<p>I asked in <a href=\"https:\/\/splunk-usergroups.slack.com\/messages\/search-help\">#search-help<\/a>.<\/p>\n<p>And within a couple minutes, had some ideas from somebody to use the &#8220;hidden field&#8221; <code>date_day<\/code> and do a <code>| stats dc(ip) by date_day site<\/code>. Unfortunately, this data source is <a href=\"https:\/\/smile.amazon.com\/s\/?tag=antipaucity-20&#038;creative=392009&#038;campaign=212361&#038;field-keywords=JSON\">JSON<\/a> that comes-in via the <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/Data\/UsetheHTTPEventCollector\">HEC<\/a>.<\/p>\n<p>Poo.<\/p>\n<div class=\"wp-block-image is-style-default\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" src=\"https:\/\/media1.tenor.com\/images\/cbf642d6a9033e5953037624dd7882f0\/tenor.gif\" alt=\"\"\/><\/figure>\n<\/div>\n<h2>Lo and behold!<\/h2>\n<p>I can &#8220;fake&#8221; <code>date_day<\/code> by using <code><a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/7.3.1\/SearchReference\/DateandTimeFunctions#strftime.28X.2CY.29\">strftime<\/a><\/code>!<\/p>\n<p>Specifically, here&#8217;s the <code><a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/SearchReference\/DateandTimeFunctions\">eval<\/a><\/code> command:<\/p>\n<p><code>| eval date=strftime(_time,\"%Y-%m-%d\")<\/code><\/p>\n<p>This converts from the hidden <code>_time<\/code> field (in Unix epoch format) to <code>yyyy-mm-dd<\/code>.<\/p>\n<h2><em><strong>This is the <img decoding=\"async\" src=\"https:\/\/s.w.org\/images\/core\/emoji\/13.0.0\/72x72\/1f511.png\" alt=\"?\" class=\"wp-smiley\" style=\"height: 1em; max-height: 1em;\" \/>!<\/strong><\/em><\/h2>\n<p>What does this line do? It lets me <code>stats<\/code>-out by <em>day<\/em> and site (just like <code>timechart<\/code> does &#8230; but it runs <em><strong>way<\/strong><\/em> faster (Why? I Don&#8217;t Know. He&#8217;s on third. And I Don&#8217;t Give a Darn! (Oh! That&#8217;s our shortstop!)).<\/p>\n<h4>How much faster?<\/h4>\n<p>At <em><strong>least<\/strong> twice as fast<\/em>! It takes ~2200 seconds to complete, but given that the <code>timechart<\/code> form was being nuked at 3600 seconds, and it was only about 70% done &#8230; this is better!<\/p>\n<p>The final form for the search:<\/p>\n<p><code>index=ndx sourcetype=srctp site=* ip=* earliest=-90d@ latest=-1d@<br \/>| table site ip _time<br \/>| eval date=strftime(_time,\"%Y-%m-%d\")<br \/>| stats dc(ip) as inventory by date site<\/code><\/p>\n<p>I&#8217;ve got this in a daily-<a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/Report\/Schedulereports\">scheduled<\/a> Report that I then draw-into <a href=\"https:\/\/docs.splunk.com\/Documentation\/Splunk\/latest\/Viz\/CreateDashboards\">Dashboard<\/a>(s) as needed (no point in running more often, since it&#8217;s summary data that only &#8220;changes&#8221; (at most) once a day).<\/p>\n<p>Hope this helps somebody &#8211; please leave a comment if it helps you!<\/p>\n<p>from antipaucity https:\/\/antipaucity.com\/2020\/08\/18\/how-to-timechart-possibly-better-than-timechart-in-splunk\/<br \/>\nvia <a href=\"https:\/\/ifttt.com\/?ref=da&#038;site=wordpress\">IFTTT<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I recently had cause to do an extensive trellised timechart for a dashboard at $CUSTOMER in Splunk. They have a couple hundred locations reporting networked devices. I needed to report on how many devices they&#8217;ve reported every day over the last 90 days (I would have liked to go back further&#8230;but retention is only 90 &hellip;<br \/><a href=\"https:\/\/merikebi.warrenmyers.com\/?p=18502\" class=\"more-link pen_button pen_element_default pen_icon_arrow_double\">Continue reading <span class=\"screen-reader-text\">how-to timechart [possibly] better than timechart in splunk<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[4],"tags":[95],"keyring_services":[],"class_list":["post-18502","post","type-post","status-publish","format-standard","hentry","category-blih","tag-antipaucity"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/18502","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=18502"}],"version-history":[{"count":1,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/18502\/revisions"}],"predecessor-version":[{"id":18503,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/18502\/revisions\/18503"}],"wp:attachment":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=18502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=18502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=18502"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fkeyring_services&post=18502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}