{"id":3249,"date":"2018-01-10T13:43:15","date_gmt":"2018-01-10T18:43:15","guid":{"rendered":"https:\/\/antipaucity.com\/?p=3249"},"modified":"2018-01-10T13:43:15","modified_gmt":"2018-01-10T18:43:15","slug":"but-i-got-them-on-sale","status":"publish","type":"post","link":"https:\/\/merikebi.warrenmyers.com\/?p=3249","title":{"rendered":"but, i got them on sale!"},"content":{"rendered":"<p>Back in August 2008, I had a one-week &#8220;quick start&#8221; professional services engagement in <a href=\"http:\/\/www.nutleynj.org\">Nutley New Jersey<\/a>. It was a supposed to be a super simple week: install <a href=\"https:\/\/antipaucity.com\/tag\/hpsa\">HP Server Automation<\/a> at BT <a href=\"https:\/\/www.glassdoor.com\/Reviews\/BT-Nutley-Reviews-EI_IE3463.0,2_IL.3,9_IC1126915.htm\">Global<\/a>.<\/p>\n<p>Another ProServe engineer was onsite to setup HP Network Automation.<\/p>\n<p>Life was gonna be easy-peasy &#8211; the <em>only<\/em> deliverable was to setup and verify a vanilla HPSA installation.<\/p>\n<p>Except, like every Professional Services engagement in history, all was not as it seemed.<\/p>\n<p>First monkey wrench: our primary technical contact \/ champion was an old-hat {{Sun Solaris}} fan (to the near-exclusion of any other OS for any purpose &#8211; he even wanted to run {{SunOS}} on his laptop).<\/p>\n<p>Second monkey wrench: expanding on the first, out technical contact was super excited about the servers he&#8217;d gotten just the weekend before from Sun because they were &#8220;on sale&#8221;.<\/p>\n<p>It&#8217;s time for a short background digression. Because <a href=\"https:\/\/antipaucity.com\/2011\/08\/08\/why-technical-intricacies-matter\/#.WlZG95M-fgo\">technical intricacies matter<\/a>.<\/p>\n<p>HP Server Automation was written on Red Hat Linux. It worked great on RHEL. But, due to some [large] customer requests, it also supported running on Sun Solaris.<\/p>\n<p>In 2007, Sun introduced a novel architecture dubbed, &#8220;Niagara&#8221;, or UltraSPARC T1, which they offered in their <a href=\"https:\/\/en.wikipedia.org\/wiki\/UltraSPARC_T1\">T1000<\/a> and T2000 series servers. Niagara did several <a href=\"http:\/\/www.cs.wm.edu\/~kemper\/cs654\/slides\/niagara.pdf\">clever<\/a> things &#8211; it offered multiple threads running per core, <a href=\"http:\/\/www.oracle.com\/technetwork\/systems\/opensparc\/2008-oct-opensparc-slide-cast-04-ds-1539002.html\">with<\/a> as many as 32 <a href=\"https:\/\/www.anandtech.com\/show\/1909\">simultaneous<\/a> processes running.<\/p>\n<p>According to AnandTech, the UltraSPARC T1 was a &#8220;72 W, 1.2 GHz chip almost 3 times (in SpecWeb2005) as fast as four Xeon cores at 2.8 GHz&#8221;.<\/p>\n<p>But there is always a tradeoff. The tradeoff Sun chose for the first CPU in the product line was to share a single FPU (floating point unit) between the integer cores and pipelines. For workloads that mostly involve static \/ simple data (ie, not much in the way of calculation), they were blazingly fast.<\/p>\n<p>But sharing an FPU brings problems when you need to actually do floating-point math &#8211; as cryptographic algorithms and protocols all end up relying upon for gathering entropy for their random value generation processes. Why does this matter? Well, in the case of HPSA, not only is all interprocess, intraserver, and interserver communication secured with <a href=\"https:\/\/antipaucity.com\/?s=ssl\">HTTPS<\/a> certificates, but because large swaths are written in Java, each JVM needs to emulate its own FPU &#8211; so not <em>only<\/em> is the single FPU shared between all of the integer cores of the T1 CPU, it is further time-sliced and shared amongst every JRE instance.<\/p>\n<p>At the time, the &#8220;standard&#8221; reboot time for a server running in an SA Core was generally benchmarked at ~15-20 minutes. That time encompassed all of the following:<\/p>\n<ul>\n<li>stop all SA processes (in the proper order)<\/li>\n<li>stop Oracle<\/li>\n<li>restart the server<\/li>\n<li>start Oracle<\/li>\n<li>start all SA components (in the proper order)<\/li>\n<\/ul>\n<p>As you&#8217;ll recall from my <a href=\"https:\/\/antipaucity.com\/2011\/08\/08\/why-technical-intricacies-matter\/#.WlZR3ZM-fgo\">article<\/a> on the Sun JRE 1.4.x from 6.5 years ago, there is a Java component (the Twist) that <em>already<\/em> takes a long time to start as it seeds its <a href=\"https:\/\/antipaucity.com\/2016\/04\/01\/improve-your-entropy-pool-in-linux\/#.WlZSXJM-fgo\">entropy<\/a> pool.<\/p>\n<p>So when it is <em>sharing<\/em> the single FPU not only between other JVMs, but between every other process which might end up needing it, the total start time is reduced <em>dramatically<\/em>.<\/p>\n<p>How dramatically? Shutdown <em>alone<\/em> was taking upwards of 20 minutes. Startup was north of 35 minutes.<\/p>\n<p>That&#8217;s right &#8211; instead of ~15-20 minutes for a full restart cycle, if you ran HPSA on a T1-powered server, you were looking at ~60+ minutes to restart.<\/p>\n<p>Full restarts, while not incredibly common, are not all that unordinary, either.<\/p>\n<p>At the time, it was not unusual to want to fully restart an HPSA Core 2-3 times per month. And during initial installation and configuration, restarts need to happen 4-5 times <strong>in addition to<\/strong> the number of times various components are restarted <em>during installation<\/em> as configuration files are updated, new processes and services are started, etc.<\/p>\n<p>What <em>should<\/em> have been about a one-day setup, with 2-3 days of knowledge transfer &#8211; turned into nearly 3 days <em>just to install and initially configure the software<\/em>.<\/p>\n<p>And why were we stuck on this &#8220;revolutionary&#8221; hardware? Because of what I noted earlier: our main technical contact was a die-hard Solaris fanboi who&#8217;d gotten these servers &#8220;on sale&#8221; (because their Sun rep &#8220;liked them&#8221;).<\/p>\n<p>How big a &#8220;sale&#8221; did he get? Well, his sales rep <em>told<\/em> him they were getting these last-model-year boxes for 20% off list <strong>plus<\/strong> an additional 15% off! That sounds pretty good &#8211; depending on how you do the math, he was getting somewhere between 32% and 35% off the list price &#8211; for a little over $14,000 a piece (they&#8217;d bought two servers &#8211; one to run {{Oracle RDBMS}} (which Oracle themselves recommended <strong>not<\/strong> running on the T1 CPU family), and the other to run HPSA proper).<\/p>\n<p>Except his sales rep lied. Flat-out lied. How do I know? Because I used Sun&#8217;s own server configurator site and was able to configure two identical servers for just a smidge over $15,000 each &#8211; with no discounts. That means they got 7% off list &#8230;<br \/>\n<strong>tops<\/strong>.<\/p>\n<p>So not <em>only<\/em> were they running hardware barely discounted off list (and, interestingly, only slightly cheaper (less than $2000) than the next generation T2-powered servers which had a single FPU per core, not per CPU (which still had some performance issues, but at least weren&#8217;t dog-vomit slow), but they were running on Solaris &#8211; which had always been a second-class citizen when it came to HPSA performance: all things being roughly equal, x86 hardware running {{RHEL}} would <em>always<\/em> smack the pants off SPARC hardware running Solaris under Server Automation.<\/p>\n<p>For kicks, I configured a pair of servers from Dell (because their online server configurator worked a lot better than any other I knew of, and because I wanted to demonstrate that just because SA was an HP product didn&#8217;t mean you had to run HP servers), and was able to massively out-spec two x86 servers for less than $14,000 a pop (more CPU cores, more RAM, more storage, etc) and present my findings as part of our write-up of the week.<\/p>\n<p>Also for kicks, I demoed SA running in a 2-CPU, 4GB VM on my <em>laptop<\/em> rebooting faster than either T1000 server they had purchased could run.<\/p>\n<p>Whats the moral of this story? There&#8217;s two (at least):<\/p>\n<ol>\n<li>Always always always find out from your vendor if they have a preferred or suggested architecture before namby-pamby buying hardware from your favorite sales rep, and<\/li>\n<li>Be ever ready and willing to kick your preconceived notions to the sidelines when presented with evidence that they are not merely ill thought out, but out and out, objectively wrong<\/li>\n<\/ol>\n<p>These are fundamental tenets of <a href=\"https:\/\/antipaucity.com\/2013\/05\/11\/automating-or-automation\/#.WlZbc5M-fgo\">automation<\/a>:<\/p>\n<blockquote><p>&#8220;Too many people try to take new tools and make them fit their current processes, procedures, and policies \u2013 rather than seeing what policies, procedures, and processes are either made redundant by the new tools, or can be improved, shortened, or \u2013 wait for it \u2013 automated!&#8221;<\/p><\/blockquote>\n<p>You <em><strong>must<\/strong> always<\/em> be reviewing and rethinking your preconceived notions, what policies you&#8217;re currently following, etc. As I <a href=\"https:\/\/wiki.securityweekly.com\/SSWEpisode68\">heard<\/a> recently, you need to <a href=\"https:\/\/www.strategy-business.com\/article\/How-to-Break-Bad-Business-Habits\">reverse your benchmarks<\/a>: don&#8217;t ask, &#8220;why are we doing X?&#8221;; ask, &#8220;what would happen if we didn&#8217;t do X?&#8221;<\/p>\n<p>That was a <a href=\"https:\/\/antipaucity.com\/2006\/09\/28\/ask-the-right-question\/#.WlZdwZM-fgo\">question<\/a> <a href=\"https:\/\/antipaucity.com\/2014\/05\/06\/what-to-automate\/#.WlZdk5M-fgp\">never<\/a> <a href=\"https:\/\/antipaucity.com\/2013\/10\/25\/what-should-a-professional-services-groups-goals-be\/#.WlZdf5M-fgp\">asked<\/a> by anyone prior to our arrival to implement what <a href=\"https:\/\/antipaucity.com\/2013\/10\/22\/unsales\/#.WlZdgpM-fgp\">sales<\/a> had sold them.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Back in August 2008, I had a one-week &#8220;quick start&#8221; professional services engagement in Nutley New Jersey. It was a supposed to be a super simple week: install HP Server Automation at BT Global. Another ProServe engineer was onsite to setup HP Network Automation. Life was gonna be easy-peasy &#8211; the only deliverable was to &hellip;<br \/><a href=\"https:\/\/merikebi.warrenmyers.com\/?p=3249\" class=\"more-link pen_button pen_element_default pen_icon_arrow_double\">Continue reading <span class=\"screen-reader-text\">but, i got them on sale!<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_hide_featured_image":false,"_jetpack_featured_image_duplicate_dismissed":0,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[7,23,41,48],"tags":[107,394,559,695,735],"keyring_services":[],"class_list":["post-3249","post","type-post","status-publish","format-standard","hentry","category-commentary","category-insights","category-technical","category-work","tag-architecture","tag-hpsa","tag-performance","tag-solaris","tag-sunos"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/3249","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3249"}],"version-history":[{"count":0,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=\/wp\/v2\/posts\/3249\/revisions"}],"wp:attachment":[{"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3249"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/merikebi.warrenmyers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fkeyring_services&post=3249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}