What are you trying to achieve?<\/p>\n
If you’re sending "unique" events to the HEC, or you’re running UFs on "unique" logs, you’ll never get duplicate "records when indexing".<\/p>\n
It sounds<\/em> like you (perhaps routinely?) resend the same data to your aggregation platform – which is not a problem with the aggregator<\/em>, but with your sending<\/em> process.<\/p>\n Almost like you’re doing a MySQL<\/a>\/PostgreSQL<\/a> "insert if not exists" operation. If that is a correct understanding of your situation, based on your statement<\/p>\n We currently use "document id" in ElasticSearch to deduplicate records when indexing: then you need to evaluate what is going "wrong" in your sending<\/em> process that you feel you need to pre-clean the data before ingesting it.<\/p>\n It is true that Splunk won’t "deduplicate records when indexing" – because it presumes<\/em> the data coming-in to be ‘correct’ from whatever is submitting it.<\/p>\n How are you getting duplicate data in the first place?<\/p>\n Fields in Splunk which begin with the underscore (eg If you really<\/em> have a problem with [lots of\/too much] duplicate data, and there is no way to fix your sending process[es]<\/strong><\/em>, then you’ll need to rely on deduplication operations<\/a> in SPL<\/a> when searching<\/a> for\/reporting on whatever you’ve ingested (primarily<\/em> by using from User warren – Stack Overflow https:\/\/stackoverflow.com\/questions\/65101933\/splunk-record-deduplication-using-an-unique-field\/65109031#65109031 What are you trying to achieve? If you’re sending "unique" events to the HEC, or you’re running UFs on "unique" logs, you’ll never get duplicate "records when indexing". It sounds like you (perhaps routinely?) resend the same data to your aggregation platform – which is not a problem with the aggregator, but with your sending …\n
\nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/docs-index_.html<\/a>
\nWe generate the id using hash of the content of the each log-record.<\/p>\n<\/blockquote>\n_time<\/code>, _cd<\/code>, etc) are not<\/strong><\/em> editable\/sendable – they’re generated by Splunk when it receives data. IOW, they’re all internal<\/em> fields. Searchable. Usable. But not overrideable.<\/p>\nstats<\/code><\/a> and, when absolutely necessary\/unavoidable, dedup<\/code><\/a>).<\/p>\n
\nvia IFTTT<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Continue reading Answer by warren for Splunk : Record deduplication using an unique field<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4],"tags":[991],"keyring_services":[],"class_list":["post-26103","post","type-post","status-publish","format-standard","hentry","category-blih","tag-stackexchange"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/26103","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=26103"}],"version-history":[{"count":1,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/26103\/revisions"}],"predecessor-version":[{"id":26104,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/26103\/revisions\/26104"}],"wp:attachment":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=26103"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=26103"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=26103"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fkeyring_services&post=26103"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}