The emergence of two-way controllable devices has opened up a wealth of possibilities for utilities and their customers. It has enabled customers to adjust the temperature of their home from thousands of miles away while giving utilities precise forecasting of demand response capacity. It has also brought technological challenges to vendors communicating with thousands of devices. At Comverge, we take great pride in IntelliSOURCE's ability to deliver large scale demand response programs for our clients and it was essential that our system deliver the same results for programs that used modern two-way devices as it did for legacy one-way paging devices. We recently completed a project to analyze how efficiently we communicate with devices and how we could optimize our performance. In this series of three blog posts, we examine the process we took to test our system, the results we obtained, and some lessons learned.
Here's a bit more background. Traditional paging networks are able to send the same single message to ten thousand devices just as simply as sending to ten devices. Besides being costly to install and maintain, the downside of those networks is that they do not provide real time feedback from the devices
. Two-way devices solve that problem and most (including all Wi-Fi devices) are IP addressable. The downside of IP addressing is that every device must be sent a unique message so controlling ten thousand devices takes 9,990 more messages than controlling ten devices.
Some demand response management systems (DRMS) work by breaking up load into large groups (treating ten thousand thermostats like a large industrial facility), determining which groups to control, then sending a proprietary or OpenADR signal to a different system that actually communicate with devices. Comverge's IntelliSOURCE coupled with our IntelliTEMP and IntelliPEAK DirectLink two-way devices has the more complex job of actually sending those messages to the ten thousand individual devices. The benefit of this approach is greatly improved forecast granularity and the flexibility for surgical event dispatch.
Determining what to test
The first question we asked was "what is our goal?" The answer, of course, depends on your exact business need. Working with our customers, we reached a goal of initiating a demand response event, selecting the devices to control, sending messages to 100,000 devices, and receiving the two-way acknowledgement in less than 120 seconds. Future testing will scale to 500,000 - 1,000,000 devices. We had a very specific and measurable goal, but it doesn't take into account the larger picture.
The Comverge IntelliSOURCE DRMS is used by more than just control room operators. Device installers use the system to manage their daily work. Customer support specialists use the system to answer customer questions. Measurement and verification specialists use the system to gather and process device telemetry. We needed to determine what other areas of the system would be in use that could have an impact on the performance.
We began by analyzing the production web logs to profile what actions were being taken by users. We also looked at the logs from the application's background processes. We concentrated on those executed around the time demand response events were called and those with longer run times. This gave us a good starting point but it was difficult to determine the performance impact of each action we were seeing in the log.
Next we used tcpdump
to capture the raw network traffic between our application and its MySQL database. We then used Percona's pt-query-digest
to analyze the captured data. This approach gave us more insight into the system's behavior than MySQL's standard logging toolset with zero downtime or impact to our customer. The results of pt-query-digest
allowed us to easily see the frequency and execution time of different database queries and to understand the impact of different system actions.
Putting this all together we came up with our test script:
- Execute a demand response event sending messages to a rotating portion of the population
- Process the two-way telemetry received from the devices
- View the real time status of devices acknowledging the demand response event
- View the real time system status and forecasted capacity
- Process multiple device installations, registrations and commissionings
Counts of production web requests used to determine test steps
Check back tomorrow for Part 2 where we'll discuss our test data and device simulation approach.
These devices also do not provide customer engagement functionality and are limited to a utility direct install channel.