Sources

Three parameters are similar for all the source components :

name
 Name of the source. A name is necessary for each component and it must be unique.
polls
 Number of polls to execute.
pollPeriod
 Time (in seconds) between each poll.
exitOnError 'y' or 'n'.
When 'n' selected, if an error occurs, the message will be saved in the directory defined in the "errPath" attribute.
errPath Directory where messages should be saved when their processing is erroneous (only if you set exitOnError to "n").

 

 

The file source component gets data from the local file system.

Parameters :

type fileSource
filePath Directory where to look for files.
fileFilter

Permits to filter files to be processed. * and ? can be used as wildcards.
(ex : foo*.txt or *d*.?ip)
newExtension If this parameter isn't empty, the processed file will have it appended to its name, in commit phase.
Example :
- file => foo.txt
- newExtension => done
- In commit phase, foo.txt will be renamed to foo.txt.done

 

 

 



The ftp source component gets data from a remote file system, accessed via FTP.

Parameters :

type ftpSource
filePath Directory where to look for files.
fileFilter

Permits to filter files to be processed. * can be used as wildcards.
(ex : foo*.txt or *d*.*ip)
newExtension If this parameter isn't empty, the processed file will have it appended to its name, in commit phase.
Example :
- file => foo.txt
- newExtension => done
- In commit phase, foo.txt will be renamed to foo.txt.done
ftpHost URL or IP adress of the FTP server
ftpPort Port of the FTP server (if empty, 21 will be used by default)
ftpUser User for FTP connection
ftpPass Password for FTP connection

 

 

 

The http source component gets data via an HTTP GET (POST isn't yet supported).

Parameters :

type HTTPSource
URL URL to call for data retrieval

 

HTTPSource can have extra parameters :

<source name="source1" type="HTTPSource" polls="3" pollPeriod="5" URL="http://localhost/test.cgi" >
+++++<params>
+++++ +++++<param name="param1" value="foo1"/>
+++++ +++++<param name="param2" value="foo2"/>
+++++ +++++<param name="param3" value="foo3"/>
+++++ +++++<param name="param4" value="foo4"/>
+++++</params>
+++++<params>
+++++ +++++<param name="param1" value="bogus1"/>
+++++ +++++<param name="param2" value="bogus2"/>
+++++ +++++<param name="param3" value="bogus3"/>
+++++ +++++<param name="param4" value="bogus4"/>
+++++</params>
</source>

Here, the source will be called as many times per poll than there are params tag. With the example above, at each poll, the following URLs will be called :

+++++ +++++http://localhost/test.cgi?param1=foo1&param2=foo2&param3=foo3&param4=foo4
+++++ +++++and
+++++ +++++http://localhost/test.cgi?param1=bogus1&param2=bogus2&param3=bogus3&param4=bogus4

The result of the second call will be appended to the one of the first call.

 

 

 

The SQLSource component outputs the result of a SQL Statement in XML or into a flat-delimited format.

Parameters :

type SQLSource
SQL SQL Statement to be executed at process stage and mapped to its XML representation
commitSQL SQL Statement to be executed after each message has been processed correctly.
Let's consider you want to delete from the source table each extracted record. The commitSQL statement owuld look like the following :
DELETE FROM table WHERE id_record = $id_record

This requires the field "id_record" to be extracted by the SQL Statement (the process one).
Each field preceded by a $ will be mapped to the value extracted by the SQL Statement (as in the SQLSink component).
nbThreads Number of threads that execute commitSQL statements in parallel
dbType The JDBC driver of your database (for example : com.mysql.jdbc.Driver)
dsn The JDBC URL (for ex : jdbc:mysql://localhost/xpweb3)
user Login to connect to db
password Password to connect to db
rootTag As the name says, name of the root tag of the output XML document
recTag Name of the tag that embraces the records in the output XML document
encoding Encoding of the output XML file
msgSize Number of records to put in each out document (the source will generate as many documents as necessary, which are processed one by one by the following sinks/pipes). Put 0 if you only want all records to be sent into the message.
outputFormat "xml" or "flat"
delimiter Ascii code of the character that will be used to separate fields (default is 59 which corresponds to ;)
newlineChar Ascii code of the character that will be used to separate records (default is 10 which corresponds to a carriage return)
header "y" or "n"
Useful only if output format is "flat". If "y" is selected, a first line will be created in the file with the name of the different fields (uses the delimiter and newlineChar defined earlier)

 

For example the following config :

<source name="SQLSource" type="SQLSource" polls="1" pollPeriod="10" SQL="SELECT field1,field2,field3,field4,field5 from reference.service" dbType="odbc" dsn="teradata_4700" user="user" password="pass" rootTag="racine" recTag="rec" encoding="UTF-8" msgSize="200" outputFormat="xml">

Will give the following output :

<?xml version="1.0" encoding="UTF-8">
<racine>
+++++<rec>
+++++ +++++<field1>value1</field1>
+++++ +++++<field2>value2</field2>
+++++ +++++<field3>value3</field3>
+++++ +++++<field4>value4</field4>
+++++ +++++<field5>value5</field5>
+++++</rec>
+++++<rec>
+++++ +++++<field1>value6</field1>
+++++ +++++<field2>value7</field2>
+++++ +++++<field3>value8</field3>
+++++ +++++<field4>value9</field4>
+++++ +++++<field5>value10</field5>
+++++</rec>
+++++<rec>
+++++ +++++<field1>value11</field1>
+++++ +++++...
+++++ +++++...
</racine>

 

 

This is an unusual component... Sometimes (well, might even be more than that) you need to get more than a bunch of rows from a table translated to XML. You need a whole structured XML document with a complex and deep architecture. The SQLTreeSource is what you need then. It permits to describe an XML tree, each node being the result of an SQL statement.
Here's an example :

<source name="SQLTreeSource" type="SQLTreeSource" dbType="odbc" dsn="localhost_mysql" user="root" password="" rootTag="XPWeb" encoding="ISO-8859-1" msgSize="0" polls="1" pollPeriod="0">
<treeQuery>
<query SQL="select * from iterations" recTag="iteration">
+<query SQL="select * from stories" parentLink="iteration_id = $id$" recTag="story">
+++<query SQL="select * from tasks" parentLink="story_id = $id$ and iteration_id = $iteration_id$" recTag="task"/>
+</query>
</query>
</treeQuery>
</source>

Here is the XML output of the source described above :

<?xml version="1.0" encoding="ISO-8859-1"?>
<XPWeb>
+++++<iteration>
++++++++++++<id>1</id>
++++++++++++<project_id>1</project_id>
++++++++++++<working_days_name>ite_1</working_days_name>
++++++++++++<load_factor>1.6</load_factor>
++++++++++++<name>07/04/2003</name>
++++++++++++<description>Mise en recette Valo
++++++++++++<date>2003-04-30</date>
++++++++++++<story>
++++++++++++++++<id>3</id>
++++++++++++++++<iteration_id>1</iteration_id>
++++++++++++++++<project_id>1</project_id>
++++++++++++++++<name>Evironnement de tests unitaires</name>
++++++++++++++++<description>Création d'un environnement de tests unitaires et de non-régression.</description>
++++++++++++++++<validation_criteria></validation_criteria>
++++++++++++++++<priority>8</priority>
++++++++++++++++<risk>3</risk>
++++++++++++++++<weight>2</weight>
++++++++++++++++<task>
++++++++++++++++++++<id>15</id>
++++++++++++++++++++<story_id>3</story_id>
++++++++++++++++++++<iteration_id>1</iteration_id>
++++++++++++++++++++<project_id>1</project_id>
++++++++++++++++++++<responsible_team_member_id>5</responsible_team_member_id>
++++++++++++++++++++<working_days_name>ite_1</working_days_name>
++++++++++++++++++++<name>Scripts de création des données (TERA)</name>
++++++++++++++++++++<description>Ecriture des scripts de création de base</description>
++++++++++++++++++++<start_date>2003-04-10</start_date>
++++++++++++++++++++<weight>3</weight>
++++++++++++++++++++<done>4.8</done>
++++++++++++++++++++<todo>0</todo>
++++++++++++++++++++<end_date>2003-04-10</end_date>
+++++++++++++++</task>
+++++++++++++++<task>
++++++++++++++++++++<id>16</id>
++++++++++++++++++++<story_id>3</story_id>
++++++++++++++++++++<iteration_id>1</iteration_id>
++++++++++++++++++++<project_id>1</project_id>
++++++++++++++++++++<responsible_team_member_id>5</responsible_team_member_id>
++++++++++++++++++++<working_days_name>ite_1</working_days_name>
++++++++++++++++++++<name>Environnement $U - Session et Uprocs</name>
++++++++++++++++++++<description>Création des sessions et uprocs propres aux tests.</description>
++++++++++++++++++++<start_date>2003-04-16</start_date>
++++++++++++++++++++<weight>1</weight>
++++++++++++++++++++<done>0</done>
++++++++++++++++++++<todo>1</todo>
++++++++++++++++++++<end_date>2003-04-16</end_date>
++++++++++++++++</task>
++++++++++</story>
++++++++++<story>
++++++++++......
++++++++++......
++++++++++......
++++++++++</story>
++++</iteration>
</XPWeb>

This is an extraction of my XPWeb database (THE Extreme Programming management tool - http://xpweb.sourceforge.net ). Three tables are accessed here : iterations, stories and tasks.
Iterations may have several stories, which may have several tasks. What I wanted here is to extract all my iterations with their stories and tasks into a single XML document.

<treeQuery>
++++ <query SQL="select * from iterations" recTag="iteration">

This permits to get all iterations at the second level (the first being the root one whose name is defined in the rootTag attribute of the source component) of the document. Each iteration will be written to XML, embraced by an <iteration> tag
Providing a new subquery this way :

<treeQuery>
++++<query SQL="select * from iterations" recTag="iteration">
++++ ++++<query SQL="select * from stories" parentLink="iteration_id = $id$" recTag="story"/>
++++</query>

Permits to define a new sublevel under iteration, writing the stories inside <story> tags.
parentLink permits to assign the stories to their iteration (we don't want all stories to be repeated under each iteration). Here, we make a link between the fields : iterations.id and stories.iteration_id
At runtime, if parentLink contains an expression that is surrounded by $, it is replaced by the value of the corresponding field of the parent query result.
At execution time, the SQL statement is modified to include the link.
(for example : select * from stories where iteration_id = 4)
Now, let's add a subquery to stories :

<treeQuery>
+<query SQL="select * from iterations" recTag="iteration">
++<query SQL="select * from stories" parentLink="iteration_id = $id$" recTag="story">
+++ <query SQL="select * from tasks" parentLink="story_id = $id$ and iteration_id = $iteration_id$" recTag="task"/>
++</query>
+</query>

Here, we've added a new sublevel, which will come under stories. It will write all tasks for each stories (with a link between stories.id and tasks.story_id).

Simple, isn't it ?

Parameters :

type SQLTreeSource
dbType The JDBC driver of your database (for example : com.mysql.jdbc.Driver)
dsn The JDBC URL (for ex : jdbc:mysql://localhost/xpweb3)
db database name (not used in JyRetic)
user Login to connect to db
password Password to connect to db
rootTag As the name says, name of the root tag of the output XML document
encoding Encoding of the output XML file (ex : UTF-8)
msgSize Number of records of the root query (iteration in the example above) to put in each out document (the source will generate as many documents as necessary, which are processed one by one by the following sinks/pipes).
Put 0 if you only want one message (with all the records) to be generated by each poll.


 

Similar to the SQLTreeSource component except that each query can be executed on a different RDBMS.
Your output XML document is then composed of data retrieved in different database servers.
Thus, each <query> tag has the following attributes (which aren't anymore on the <source> tag) :

dbType The JDBC driver of your database (for example : com.mysql.jdbc.Driver)
dsn The JDBC URL (for ex : jdbc:mysql://localhost/xpweb3)
db useless...
user Login to connect to db
password Password to connect to db

 

xmlBlaster is an open source MOM.
For more information about it, see http://www.xmlblaster.org.

This component uses pyBlaster (included and ready to use in the release) to permit the use of the synchronous get access method to xmlBlaster. Messages are deleted from the Topic at commit phase, after full processing.

Here is what it must look like in the XML config file :

<source name="xmlBlasterSource" type="xmlBlasterSource" url="http://192.168.185.206:8080" user="adi2" password="pass" polls="1" pollPeriod="0" >
++++<params>
++++ ++++<param name="key" value="&lt;key oid='' queryType='XPATH'&gt;/xmlBlaster/key[starts-with(@oid,'xml')]&lt;/key&gt;" />
++++ ++++<param name="qos" value="&lt;qos&gt;&lt;/qos&gt;" />
++++</params>
</source>

The params permit to define the key to search for message and the qos (Quality Of Service).
It is important to replace < by &lt; and > by &gt; in the key and qos values in order to have a well-formed XML file when you write your adaptor by hand ( Retic designer handles the replacement so when using it, feel free to write regular tags : <key... ).

Parameters :

type xmlBlasterSource
user Login to connect to xmlBlaster server
password Password to connect to xmlBlaster server

url

 url of the XMLRPC listener of the xmlBlaster server

 

 


This component permits to receive text messages from JMS compliant MOMs.
Both queue and topic modes are supported.

Parameters :

type JMSSource
jndiFactory JNDI Factory (ex : com.sun.jndi.fscontext.RefFSContextFactory )
jmsFactory JMS Factory (ex : JmsTopicConnectionFactory )
queueName Queue or Topic name
providerUrl URL of the JMS Provider (ex : rmi://localhost:1099/ )
subscriberType "durable" or "nondurable"
mode "queue" or "topic"
username Username for the JMS provider connection
password password for the JMS provider connection
selector Properties message selector

 


This components permits to use XPath query results on Xindice collections as data source.

Parameters :

type JMSSource
jndiFactory JNDI Factory (ex : com.sun.jndi.fscontext.RefFSContextFactory )
jmsFactory JMS Factory (ex : JmsTopicConnectionFactory )
queueName Queue or Topic name
providerUrl URL of the JMS Provider (ex : rmi://localhost:1099/ )
subscriberType "durable" or "nondurable"
mode "queue" or "topic"
username Username for the JMS provider connection
password password for the JMS provider connection
selector Properties message selector