##############
Data Pipelines
##############




************
Introduction
************

.. raw:: html

	<p>The IoT Gateways tab provides an overview of all the incoming data streams to the 4PointX IAoT Platform. Each data stream is called a Pipeline and it can be ingesting various <em>Type of Data</em> (condition, process etc.) from a variety of <em>Data Sources</em> for a Plant.</p>

.. raw:: html

	<img src='/docs/_static/media/Screenshot 2023-11-30 at 12.55.20 PM.png'>

.. raw:: html

	<p>The tab provides the following overview information.</p>

#. Total count of IoT Gateways and how many have Heartbeat
#. Total count of Tags being ingested
#. Total Samples collected and their trend
#. A tabular list of all IoT Gateways along with their details. The following actions can be performed on an IoT Gateway.
	#. Edit
	#. Start/Stop
	#. Delete
	#. Reboot
	#. Factory Reset
	
	
	



.. raw:: html

	<p><em><span style="color: rgb(151,160,175);">Reboot</span></em><span style="color: rgb(151,160,175);"> and </span><em><span style="color: rgb(151,160,175);">Factory Reset</span></em><span style="color: rgb(151,160,175);"> are available only if </span><em><span style="color: rgb(151,160,175);">Data Source</span></em><span style="color: rgb(151,160,175);"> is </span><em><span style="color: rgb(151,160,175);">Edge IoT Gateway</span></em><span style="color: rgb(151,160,175);">.</span></p>

.. note::

	A Sample is a reading for one Tag collected at an interval set using the Sampling Rate


.. raw:: html

	<p>Use <em>Create New Gateway</em> button to create a new IoT Gateway. </p>

.. raw:: html

	<img src='/docs/_static/media/Screenshot 2023-11-30 at 1.16.13 PM.png'>

.. raw:: html

	<p>The following <em>Type of Data</em> and the list of <em>Data Sources</em> and <em>Source Sub Types</em> are currently supported. The details of onboarding this data is discussed in the next section.</p>

.. raw:: html

	<p><strong>Type of Data</strong></p>

#. Condition
#. Process
#. Energy
#. Production
#. Other



.. raw:: html

	<p><strong>Data Sources </strong>and<strong> Source Sub Types</strong></p>

#. Edge IoT Gateway
	#. Modbus RTU
	#. Modbus TCP
	#. OPC-UA
	#. Analog
	#. File
	
	
	
#. Third-Party Apps
	#. Infinite Uptime
	#. CimCon Digital
	
	
	
#. Cloud
	#. AWS S3
	#. Google Cloud Storage
	#. Azure Data Lake
	
	
	



***************
Data Onboarding
***************


Edge IoT Gateway
================

.. raw:: html

	<p><em>Data Source = Edge IoT Gateway</em></p>

.. raw:: html

	<p>Please see this section for the initial configuration steps to be performed on <em>Edge IoT Gateway</em> before using the device for data onboarding.</p>


Modbus RTU
----------

.. raw:: html

	<p><em>Source Sub Type = Modbus RTU</em></p>


OPC-UA
------

.. raw:: html

	<p><em>Source Sub Type = OPC-UA</em></p>


OPC Server > Python (Producer) > Kafka > Python (Consumer) > Elasticsearch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. raw:: html

	<p>The following systems/ software can be onboarded using OPC-UA:</p>

#. Historians (provided OPC-UA Server license is installed)
#. KepServer :question_mark:







Modbus TCP
----------

.. raw:: html

	<p>…</p>


File
----

.. raw:: html

	<p><em>Source Sub Type = File</em></p>

.. raw:: html

	<p>This section describes how to onboard data from CSVs and other flat files into the 4PointX IAoT Platform.</p>

.. raw:: html

	<p>The typical data flow is:</p>

.. raw:: html

	<blockquote><p><strong>File &gt; Logstash (Producer) &gt; Kafka &gt; Python (Consumer) &gt; Elasticsearch</strong></p></blockquote>

.. raw:: html

	<p>The flat files will be present in the customer’s environment usually copied to a machine which we call Edge Server. We run the Logstash in Edge Server to tail these files and continuously ingest them into the 4PointX IAoT Workbench.</p>


Steps to onboard data from File
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

#. Prepare your data as per the template. Use these steps to test the transformed data for errors. Use the appropriate template. Do not change the order and name of columns in the template. If your data does not have information for a particular column, leave the rows blank but do keep the column header.
	#. For Process, Condition or Energy data: Use csv_template_process_condition_energy.csv
	#. For Production data: Use csv_template_production.csv
	
	
	
#. Create a new Pipeline in Configuration > Data Pipelines tab
	#. Select the Type of Data, Site, Plant and Function
	#. Select Data Source as ‘Edge IoT Gateway’ and Source Sub Type as ‘File’
	#. Make note of the pipeline_id. We need it in step 3.iii below. We will be adding pipeline_id to All Pipelines table soon. For now, get it from Discover or Dev Tools
	
	
	
#. Start the Logstash producer in the Edge Server
	#. Install Logstash if it is not installed already
	#. Use the appropriate .conf template and change the following settings inside the template
		#. path: absolute path to the folder where flat files prepared in step 1 are stored (e.g., /home/4px/plant/function/condition_data/\*.csv)
		#. sincedb_path: absolute path to the file that receives the runtime logs (e.g., /home/4px/plant/function/condition_data/pipeline.log)
		#. topic_id: pipeline_id
		#. bootstrap_servers: the IP of kafka-broker
		
		
		
	#. Start the Logstash producer using the following shell command:
		.. code-block:: text
		
			sudo /usr/share/logstash/bin/logstash -r -f <path_to_.conf_file>
		
		
	
	
	
#. Come back to Configuration > Data Pipelines tab and start the Pipeline that was created in step 2
#. You can confirm whether the data is flowing by checking the Total Events and Last Event columns in All Pipelines table. You can also confirm in the Discover by filtering for the required pipeline_id.




Installing Logstash
^^^^^^^^^^^^^^^^^^^


CentOS
""""""

.. code-block:: text

	sudo yum install java


.. code-block:: text

	sudp rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch


.. code-block:: text

	sudo tee /etc/yum.repos.d/elasticsearch.repo <<EOF
	[elasticsearch-7.x]
	name=Elasticsearch repository for 7.x packages
	baseurl=https://artifacts.elastic.co/packages/7.x/yum
	gpgcheck=1
	gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
	enabled=1
	autorefresh=1
	type=rpm-md
	EOF


.. code-block:: text

	sudo yum install --assumeyes logstash



Testing Transformed Data
^^^^^^^^^^^^^^^^^^^^^^^^

.. raw:: html

	<p>Once you prepare the data in the required template format, use the following steps to make sure it doesn’t contain any gaps or errors.</p>

#. The columns should be arranged according to the data_type template attached below.
#. the event_timestamp column should be in this format:
	.. code-block:: text
	
		%d/%m/%Y %H:%M
		
		d: day
		m: month
		Y: year
		H: hour
		M: minute
		
		Example: 20/05/2022 14:21
	
	
	.. raw:: html
	
		<p> </p>
	
#. Ensure that there are no NaN (null) values in event_timestamp, tag__name and device_identifier columns.



.. raw:: html

	<a href='/docs/_static/media/UNKNOWN_ATTACHMENT' class='media-group'><div class='image-preview'><svg width="24" height="24" viewBox="0 0 24 24" role="presentation"><path fill="#758195" fill-rule="evenodd" d="M12 4H8a2 2 0 00-2 2v12a2 2 0 002 2h8a2 2 0 002-2v-6.995h-2V18H8V6h4v2a2 2 0 002 2h3.5a.5.5 0 00.5-.5V8.213a.5.5 0 00-.145-.352L14.61 4.59A2 2 0 0013.19 4H12zM3 0h18a3 3 0 013 3v18a3 3 0 01-3 3H3a3 3 0 01-3-3V3a3 3 0 013-3z"></path></svg></div><div class='filename'>UNKNOWN_ATTACHMENT</div></a>


Third-Party Apps
================


Infinite Uptime
---------------

.. raw:: html

	<p>…</p>


CimCon Digital
--------------

.. raw:: html

	<p>…</p>

.. raw:: html

	<p> </p>


Cloud
=====


AWS S3
------

.. raw:: html

	<p>…</p>

.. raw:: html

	<p> </p>


Google Cloud Storage
--------------------

.. raw:: html

	<p>…</p>


Azure Data Lake
---------------

.. raw:: html

	<p>…</p>

.. toctree::

	iot_gateway/index