Accelerating Grid-Edge Analytics utilizing COMTRADE Recordsdata with Apache Spark

This answer accelerator and weblog had been created in collaboration with Schneider Electrical. We might wish to thank Dan Sabin, a Schneider Electrical Distinguished Technical Skilled and secretary of the IEEE/IEC Twin Emblem Upkeep Crew targeted on the revision of the COMTRADE-2013 commonplace for lending his experience.


Steady, dependable electrical energy era and supply is important for our trendy lifestyle. However energy grids are continuously evolving, extra so now than within the first 100 years of their existence, making it important that energy mills adhere to established requirements. Via adherence to requirements, the transmission and distribution traces, substations, transformers and different elements that comprise the grid work in live performance to ship predictable and top quality energy to the patron.

To make sure energy supply adheres to requirements, Clever Digital Gadgets (IEDs) together with safety relays, digital fault recorders, phasor measuring items (PMUs), and energy high quality displays are deployed in electrical substations to carefully monitor the traits of the ability flowing by the grid. These IEDs quickly pattern and/or derive electrical portions like voltage, present, energy, frequency, part angle, rms values, harmonics, and extra. When irregular circumstances are detected, like voltage sags or electrical faults, the IEDs bundle the measurements taken simply previous to and throughout the anomalous occasion as industry-standard COMTRADE recordsdata. These recordsdata are a vital ingredient of diagnosing and correcting the circumstances that result in such occasions.

The COMTRADE file format was first specified by the Institute of Electrical and Electronics Engineers in IEEE Std C37.111 in 1991 as a “Frequent Format for Transient Information Trade for Energy Techniques”. IEEE accredited revisions in 1999 and 2013 to take care of the format’s relevance within the face of evolving technical necessities. The 2013 version was accomplished in collaboration with the Worldwide Electrotechnical Fee, the place it was revealed concurrently as IEC 60255-24:2013.

In the present day, a wide range of specialised software program packages assist the evaluation of COMTRADE formatted recordsdata however these software program packages sometimes course of the info in a single course of. As extra practitioners within the discipline of electronics and electrical engineering discover using machine studying, and because the quantity of knowledge continues to develop, there’s a rising want for assist of COMTRADE knowledge in distributed computing techniques like Apache Spark.

Understanding COMTRADE Recordsdata

Many organizations within the electrical engineering house have collected huge troves of COMTRADE knowledge. To report a single occasion, the COMTRADE format specifies the creation of a compulsory configuration (.cfg) file and a compulsory knowledge (.dat) file inside which higher-level particulars affecting the interpretation of the assorted readings and the readings themselves, respectively, are recorded. Extra recordsdata could also be included within the assortment of recordsdata surrounding a selected electrical disturbance (e.g. header recordsdata and data recordsdata), however these are elective and never at all times captured.

The variable construction of those recordsdata necessitates the inclusion of a configuration file. Moreover, it could be essential to learn a number of measurements saved in separate COMTRADE recordsdata spanning many seconds, minutes and even hours. These challenges mix to make the studying of COMTRADE recordsdata a formidable job for general-purpose knowledge and analytics platforms. Specialised libraries, just like the open supply comtrade library in Python, have emerged to help organizations with the studying of particular person units of knowledge, however studying giant volumes of those recordsdata can nonetheless be difficult.

Utilizing COMTRADE Recordsdata with Apache Spark

Fortunately, trendy analytics platforms, like Apache Spark, are extremely extensible. Via a user-defined operate, knowledge from the assorted recordsdata will be learn utilizing a specialised library and a unified set of knowledge outputs will be returned. With assist for advanced knowledge buildings, this knowledge will be returned in a fashion that respects the connection between higher-level configuration parts and lower-level readings, whereas nonetheless permitting analysts the flexibility to simply entry parts from totally different elements of the info construction.

Whereas this knowledge will be persevered in one thing extra aligned with the native COMTRADE knowledge buildings, many analysts might discover that additional simplifications, such because the separation of configuration particulars into one desk and lower-level readings into one other, can simplify the retrieval of knowledge to be used in machine studying workout routines with out impacting the integrity of the data itself. The adoption of this sample permits organizations to make these knowledge recordsdata extra broadly obtainable to a broader array of analysts and knowledge scientists, permitting them to discover new and novel evaluation approaches.

Placing It Into Follow

To show this sample, we’ve got developed a answer accelerator targeted on using knowledge within the COMTRADE format. Utilizing a big assortment of COMTRADE formatted recordsdata, we show how these could also be accessed and processed at scale utilizing Apache Spark operating within the Databricks platform. The info extracted from these recordsdata is persevered to a small set of easy-to-access tables, and the info in these tables is leveraged in coaching a easy fault detection mannequin utilizing a convolutional neural community method that has been popularized as of late by electrical engineers working together with knowledge scientists. It’s our hope that organizations targeted on electrical energy era, energy supply, energy high quality, and energy utilization will have the ability to take patterns demonstrated on this accelerator and undertake them to allow better use of their COMTRADE data belongings.

Obtain now

Related Articles


Please enter your comment!
Please enter your name here

Latest Articles