{"id":11089,"date":"2026-09-25T16:03:20","date_gmt":"2026-09-25T16:03:20","guid":{"rendered":"https:\/\/schindlerengineering.com\/public\/?p=11089"},"modified":"2026-09-25T16:10:09","modified_gmt":"2026-09-25T16:10:09","slug":"every-bug-we-shipped-this-week-and-the-0-1-kwh-that-lied-to-me","status":"publish","type":"post","link":"https:\/\/schindlerengineering.com\/public\/2026\/09\/25\/every-bug-we-shipped-this-week-and-the-0-1-kwh-that-lied-to-me\/","title":{"rendered":"Every bug we shipped this week (and the 0.1 kWh that lied to me)"},"content":{"rendered":"<p>We have been shipping an app for this solar system at a fairly indecent pace.<br \/>\nHere is every bug we found doing it, including the two where the software was<br \/>\nconfidently lying to me, and the rule change that came out of it.<\/p>\n<h2>First, the new rule: the counter only ticks over after a full kilowatt-hour<\/h2>\n<p>The front screen has a <strong>DAYS OFF THE GRID<\/strong> counter. It said<br \/>\n<strong>6<\/strong>. It should have said 13.<\/p>\n<p>On September 19th the system pulled <strong>0.1 kWh<\/strong> from PG&amp;E. A tenth of a<br \/>\nkilowatt-hour. A momentary handshake, a relay doing relay things. And that tickle reset a<br \/>\nthirteen-day run to zero.<\/p>\n<p>That is a bad rule. A tenth of a kilowatt-hour is not <em>buying electricity<\/em>. So now<br \/>\na day only breaks the streak if it draws <strong>at least 1 kWh<\/strong>.<\/p>\n<p>But here is the part I insisted on: <strong>the tickle still gets printed.<\/strong> Right<br \/>\nunder the big number it says <em>&#8220;since then: 0.1 kWh on 2026-09-19 &mdash; under the 1 kWh<br \/>\nbar&#8221;<\/em>. Generous headline, honest footnote. A dashboard that quietly swallows inconvenient<br \/>\ndata is a dashboard you stop trusting, and then what is it for?<\/p>\n<h2>The bug that hid 3.6 kilowatts<\/h2>\n<p><img decoding=\"async\" src=\"https:\/\/schindlerengineering.com\/public\/wp-content\/uploads\/2026\/09\/bug-sampling-2.png\" alt=\"Sampling resolution bug\" style=\"max-width:100%;height:auto\" \/><\/p>\n<p>I looked at my own graph and it told me the day had peaked at <strong>13.9 kW<\/strong>.<br \/>\nI had stood there and watched the thing do <strong>16 and change<\/strong>. One of us was wrong.<\/p>\n<p>Green is the same day, same data, read properly: <strong>17.4 kW<\/strong>.<\/p>\n<p>The sampler was stepping through the day in 30 minute jumps, but each call to the inverter<br \/>\nportal only covers about ten minutes. So it never looked at two thirds of the day. Worse, for<br \/>\nevery call it kept only the single reading nearest the minute it had asked about and<br \/>\n<em>threw away the rest it had already been handed<\/em>.<\/p>\n<p>Same day. Same data. <strong>3.6 kW of difference<\/strong>, invented entirely by how often<br \/>\nwe bothered to look.<\/p>\n<h3>And here is why it survived so long<\/h3>\n<p>I went back and ran the old algorithm against the new one across five days:<\/p>\n<ul>\n<li>Sep 19 &mdash; 17.3 actual, 17.1 reported. <strong>0.2 kW missed.<\/strong><\/li>\n<li>Sep 20 &mdash; 17.4 actual, 13.9 reported. <strong>3.6 kW missed.<\/strong><\/li>\n<li>Sep 22 &mdash; 17.0 actual, 16.9 reported. <strong>0.1 kW missed.<\/strong><\/li>\n<li>Sep 23 &mdash; 16.6 actual, 16.6 reported. <strong>Nothing missed at all.<\/strong><\/li>\n<li>Sep 24 &mdash; 18.1 actual, 16.8 reported. <strong>1.3 kW missed.<\/strong><\/li>\n<\/ul>\n<p>Four days out of five, the broken code looked <em>fine<\/em>. It only mangled the answer when<br \/>\nthe peak happened to fall in one of its blind spots. That is the nastiest kind of bug: not one<br \/>\nthat fails, one that is <strong>usually right<\/strong>. You cannot catch it by glancing at it.<br \/>\nYou catch it because a human who was standing in the room says &#8220;that is not what I saw.&#8221;<\/p>\n<h2>And the one where the cloud was lying about being fresh<\/h2>\n<p><img decoding=\"async\" src=\"https:\/\/schindlerengineering.com\/public\/wp-content\/uploads\/2026\/09\/vrm-freshness-comparison.png\" alt=\"Data freshness by channel\" style=\"max-width:100%;height:auto\" \/><\/p>\n<p>Red is Victron&#8217;s HTTP endpoint, which hands you a new reading about every 105 seconds no<br \/>\nmatter how fast you ask. Green is the real-time channel sitting right there the whole time at<br \/>\n1 to 2 seconds. Full write-up is in the earlier post.<\/p>\n<h2>The full list<\/h2>\n<ul>\n<li><strong>Counters that looked alive but were frozen.<\/strong> The data-age numbers only<br \/>\nupdated on the 30 second refresh, so the screen flashed black and the numbers jumped. Rebuilt<br \/>\nas self-ticking views that count up every second on their own.<\/li>\n<li><strong>Victron tab took forever.<\/strong> It waited for a 24 hour history query before<br \/>\ndrawing anything. Now the live numbers paint immediately and the charts fill in behind.<\/li>\n<li><strong>Sampling blindness.<\/strong> The 3.6 kW above.<\/li>\n<li><strong>Hard-coded battery capacity.<\/strong> 500 Ah baked into the app; the bank is now<br \/>\n1000. It is readable straight off the shunt, so now it is read straight off the shunt. Change<br \/>\nit on the shunt and the app follows.<\/li>\n<li><strong>Yesterday&#8217;s chart with today&#8217;s total.<\/strong> The energy endpoint silently<br \/>\n<em>ignores<\/em> the date you hand it and always answers for today. Browsing back a day drew<br \/>\nthat day&#8217;s curve beside today&#8217;s kilowatt-hours.<\/li>\n<li><strong>A race when flicking between days.<\/strong> Each tap launched a fetch about ninety<br \/>\ncalls deep; whichever finished last won, so you got a day you had not asked for. Every render<br \/>\nnow carries a generation number and stale work throws itself away.<\/li>\n<li><strong>A hard-coded &#8220;TODAY&#8221;.<\/strong> The heading cheerfully said TODAY above whichever<br \/>\nday you had picked.<\/li>\n<li><strong>The 0.1 kWh tickle.<\/strong> Above.<\/li>\n<\/ul>\n<h2>Two that were not the app at all<\/h2>\n<ul>\n<li><strong>My screen kept going black<\/strong> and no Windows setting would stop it. A<br \/>\nkeep-awake script from days earlier was <em>still running the old copy of itself<\/em> &mdash;<br \/>\nPowerShell reads a script once at launch, so editing the file changed nothing. Two orphans sat<br \/>\nthere blanking my monitor on a five minute timer, running elevated so I could not even kill<br \/>\nthem normally.<\/li>\n<li><strong>I DDoS&#8217;d my own Cerbo.<\/strong> Chasing the real-time feed we sent the gentlest<br \/>\nrequest in the protocol &mdash; a keepalive &mdash; every five seconds. Each one made the GX<br \/>\nre-publish <em>all 234 of its topics<\/em>. VictronConnect slowed to a crawl and my SmartShunt<br \/>\nvanished from the device list. I thought I had killed the shunt. I had just been extremely<br \/>\npolite at it, very fast, for ten minutes.<\/li>\n<\/ul>\n<h2>The pattern<\/h2>\n<p>Look at what nearly all of these have in common. Not one threw an error. The app never<br \/>\ncrashed. Nothing went red. They all just <strong>quietly reported a number that was<br \/>\nwrong<\/strong> &mdash; a stale age, a flattened peak, yesterday&#8217;s chart, today&#8217;s total, a reset<br \/>\ncounter.<\/p>\n<p>Which is why almost every fix ends the same way: <em>show the operator what the machine<br \/>\nactually knows.<\/em> Print the data&#8217;s real age and let it tick. Say which day you are looking<br \/>\nat. Show the tickle under the streak. A confident wrong number is worse than no number,<br \/>\nbecause you will go and make decisions with it.<\/p>\n<p style=\"letter-spacing:0.4em\"><strong>O N W A R D<\/strong><\/p>\n<hr>\n<p><em>Posted by my claude instance, which wrote most of these bugs and then had to go find<br \/>\nthem again. This post itself was corrected after publishing: the first version illustrated the<br \/>\nsampling bug with the wrong day and overstated it. The figures above are measured, and the<br \/>\nfive-day table is there so you can see how often the broken version looked fine.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We have been shipping an app for this solar system at a fairly indecent pace. Here is every bug we found doing it, including the two where the software was confidently lying to me, and the rule change that came out of it. First, the new rule: the counter only ticks over after a full &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/schindlerengineering.com\/public\/2026\/09\/25\/every-bug-we-shipped-this-week-and-the-0-1-kwh-that-lied-to-me\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Every bug we shipped this week (and the 0.1 kWh that lied to me)&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-11089","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/posts\/11089","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/comments?post=11089"}],"version-history":[{"count":3,"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/posts\/11089\/revisions"}],"predecessor-version":[{"id":11093,"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/posts\/11089\/revisions\/11093"}],"wp:attachment":[{"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/media?parent=11089"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/categories?post=11089"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/schindlerengineering.com\/public\/wp-json\/wp\/v2\/tags?post=11089"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}